Patent ID: 12200446

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations and other implementations are possible. For example, substitutions, additions or modifications may be made to the components illustrated in the drawings, and the illustrative methods described herein may be modified by substituting, reordering, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims.

FIG.1illustrates a user100wearing a pair of earphones110. As shown, an earphone110may be worn in the ear of user100. For example, earphone110may include a speaker device (not shown inFIG.1) configured to deliver audio to the ear of user100. Earphone110may include a microphone or other audio sensors for capturing audio signals from the environment of user100. These audio signals may be analyzed, conditioned, and/or otherwise processed and provided to the ear of user100. Accordingly, earphone110may provide an improved hearing experience for user100, by providing processed audio signals to user100, which may be easier for user100to understand. In some embodiments, earphone110may provide notifications, alerts, or other information to user100, as described further herein.

In some embodiments, earphone110may communicate wirelessly or via a wire with a computing device120. In some embodiments, computing device120may include, for example, a smartphone, or a tablet, or a dedicated processing unit, which may be portable (e.g., can be carried in a pocket of user100). Although shown inFIG.1as an external device, in some embodiments, computing device120may be provided as part of earphone110. In some embodiments, computing device120may be included in an augmented reality display device or optical head mounted display. In other embodiments, computing device120may be provided as part of another wearable or portable earphone of user100including a wrist-strap, a multifunctional watch, a button, a clip-on, etc. In yet other embodiments, computing device120may be provided as part of another system, such as an on-board automobile computing or navigation system. A person skilled in the art can appreciate that different types of computing devices and arrangements of devices may implement the functionality of the disclosed embodiments. Accordingly, in other implementations, computing device120may include a Personal Computer (PC), laptop, an Internet server, etc.

FIG.2is a schematic illustration of an exemplary system200including an earphone110(or multiple earphones110), worn by user100, and an optional computing device120and/or a server250capable of communicating with earphone110via a network240, consistent with disclosed embodiments. In some embodiments, earphone110may capture and analyze audio and image data, as described in further detail below. Earphone110may generate a processed audio signal to present to the ear of user100based at least in part on the analysis of the audio and image data. In some embodiments, earphone110may perform various other actions, which may include providing feedback or other information to a user100based on the analysis of the audio and image data. In some embodiments, optional computing device120and/or server250may provide additional functionality to enhance interactions of user100with his or her environment, as described in greater detail below.

In some embodiments, earphone110may be associated with an earphone case130, which may be configured to contain one or more earphone110. Earphone case130may perform additional functions, such as providing a charge to earphone110. In some embodiments, earphone case130may perform various processing functions for audio and/or image data, which may be captured using earphone110. Alternatively or additionally, earphone case130may capture its own audio and/or image data, as described in further detail below.

As indicated inFIG.2, the various components of system200may communicate either through direct communications, or over a network240. Network240may be a shared, public, or private network, may encompass a wide area or local area, and may be implemented through any suitable combination of wired and/or wireless communication networks. Network240may further comprise an intranet or the Internet. In some embodiments, network240may include short range or near-field wireless communication systems for enabling communication between earphone110and computing device120provided in close proximity to each other, such as on or near a user's person, for example. Earphone110may establish a connection to network240autonomously, for example, using a wireless module (e.g., Wi-Fi, cellular). In some embodiments, earphone110may use the wireless module when being connected to an external power source, to prolong battery life. Further, communication between earphone110and server250may be accomplished through any suitable communication channels, such as, for example, a telephone network, an extranet, an intranet, the Internet, satellite communications, off-line communications, wireless communications, transponder communications, a local area network (LAN), a wide area network (WAN), and a virtual private network (VPN).

As shown inFIG.2, earphone110may transfer or receive data to/from server250via network240. In the disclosed embodiments, the data being received from server250and/or computing device120may include numerous different types of information based on image data captured by earphone110or computing device120, and analysis thereof, including information related to a commercial product, or a person's identity, an identified landmark, and any other information capable of being stored in or accessed by server250. In some embodiments, data may be received and transferred via computing device120. Server250and/or computing device120may retrieve information from different data sources (e.g., a user specific database or a user's social network account or other account, the Internet, and other managed or accessible databases) and provide information to earphone110related to the analyzed image data and a recognized trigger according to the disclosed embodiments. In some embodiments, calendar-related information retrieved from the different data sources may be analyzed to provide certain time information or a time-based context for providing certain information based on the analyzed image data.

FIG.3is a block diagram illustrating the components of earphone110according to an example embodiment. As shown inFIG.3, earphone110may include a processor310, an audio sensor340, an image sensor330, a wireless transceiver360, a hearing interface350, a memory320, a power source370, and a power interface372. Earphone110is not limited to the components shown inFIG.3and may include other of fewer components depending on the application. For example, in some embodiments earphone110may also include buttons, other sensors such as a proximity sensor or similar sensors, and inertial measurements devices such as accelerometers, gyroscopes, magnetometers, temperature sensors, color sensors, light sensors, etc. Earphone110may further include a data port with suitable interfaces for connecting with an external device (not shown). In some embodiments, earphone110may not include an image sensor and/or an audio sensor.

Processor310, depicted inFIG.3, may include any suitable processing device. The term “processing device” includes any physical device having an electric circuit that performs a logic operation on input or inputs. For example, processing device may include one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logic operations. Processor310may be configured to generate a processed audio signal based on analysis of a captured audio signal or audio signal received from another source, or a captured image, as described throughout the present disclosure. For example, this may include executing instructions stored in memory320to perform selective conditioning of audio signals, analyze lip movements or other gestures and/or voice signatures, or various other analyses described above with respect to processor110. The instructions executed by the processing device may, for example, be pre-loaded into a memory integrated with or embedded into the processing device or may be stored in a separate memory (e.g., memory320).

In the embodiment illustrated inFIG.3, earphone110includes one processing device (e.g., processor310). However, in some embodiments, earphone110may include more than one processing device. Each processing device may have a similar construction, or the processing devices may be of differing constructions that are electrically coupled, connected, or disconnected from each other. For example, the processing devices may be separate circuits or integrated in a single circuit. When more than one processing device is used, the processing devices may be configured to operate independently or collaboratively. The processing devices may be coupled electrically, magnetically, optically, acoustically, mechanically or by other means that permit them to interact.

Memory320may comprise a Random Access Memory (RAM), a Read-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed, or volatile memory, or any other mechanism capable of storing instructions. In some embodiments memory320may store audio data or image data (e.g., images, videos) captured from the environment of user100. In addition, memory320may store information specific to user100, such as voice signatures, also referred to as voice prints of one or more individuals, image representations of known individuals, favorite products, personal items, and calendar or appointment information, etc. In some embodiments, processor310may determine, for example, which type of voice data or image data to store based on available storage space in memory320. In another embodiment, processor310may extract information from the image data stored in memory320.

As discussed above, earphone110may include an image sensor330for capturing image data. The term “image sensor” refers to a device capable of detecting and converting optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. The electrical signals may be used to form an image or a video stream (i.e., image data) based on the detected signal. The term “image data” includes any form of data retrieved from optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums. Examples of image sensors may include semiconductor charge-coupled devices (CCD), active pixel sensors in complementary metal-oxide-semiconductor (CMOS), or N-type metal-oxide-semiconductor (NMOS, Live MOS). In some cases, image sensor330may be part of a camera included in earphone110. In some cases, image sensor330may be part of a camera included in the same housing as earphone110. Although image sensor330is described generally herein as optic camera, it is to be understood that image sensor330is not limited to this form of capturing device, and any sensor that outputs information about the structure or appearance of the environment may be used, such a video camera, a still camera, a wide angle camera, a narrow angle camera, a thermal camera, a radar device, a Simultaneous Localization and Mapping (SLAM) device, or various other forms of sensors. In some embodiments, the camera may be a black and white camera, which may have advantages such as a better signal-to-noise (SNR) ratio, a better resolution, a simpler lens or other components, or the like.

In some embodiments, processor310may control image sensor330to capture image data and for analyzing the image data according to the disclosed embodiments. As discussed above, processor310may perform logic operations on one or more inputs of image data and other data according to stored or accessible software instructions providing desired functionality. In some embodiments, processor310may also control feedback outputting unit380to provide feedback to user100including information based on the analyzed image data and the stored software instructions.

In another embodiment, processor310may change the aiming direction of image sensor330. For example, in some embodiments, the aiming direction of image sensor330may not coincide with the field-of-view of user100. Processor310may recognize certain situations from the analyzed image data and adjust the aiming direction of image sensor330to capture relevant image data. For example, in one embodiment, processor310may detect an interaction with another individual and sense that the individual is not fully in view, because image sensor330is tilted down. Responsive thereto, processor310may adjust the aiming direction of image sensor330to capture image data of the individual. Other scenarios are also contemplated where processor310may recognize the need to adjust an aiming direction of image sensor330.

In some embodiments, processor310may communicate data to feedback-outputting unit380, which may include any device configured to provide information to a user100. Feedback outputting unit380may include one or more feedback systems for providing the output of information to user100. In the disclosed embodiments, the audible or visual feedback may be provided via any type of connected audible or visual system or both. Feedback of information according to the disclosed embodiments may include audible feedback to user100(e.g., using a Bluetooth™ or other wired or wirelessly connected speaker, or a bone conduction headphone). Feedback outputting unit380of some embodiments may additionally or alternatively produce a visible output of information to user100, for example, through a display450provided as part of computing device120, which may include an onboard automobile heads up display, an augmented reality device, a virtual reality device, a smartphone, PC, table, etc. Feedback outputting unit380may be provided as part of earphone110(as shown) or may be provided external to earphone110and communicatively coupled thereto. Feedback-outputting unit380may be configured to output visual or nonvisual feedback based on signals received from processor310, such as when processor310recognizes a hand-related trigger or other events or objects in the analyzed image data.

The term “feedback” refers to any output or information provided in response to processing at least one image in an environment. In some embodiments, feedback may include an audible or visible indication of time information, detected text or numerals, the value of currency, a branded product, a person's identity, the identity of a landmark or other environmental situation or condition including the street names at an intersection or the color of a traffic light, etc., as well as other information associated with each of these. Accordingly, processor310may be configured to detect any of these various elements within the environment of user100. In some embodiments, feedback may include an audible tone, a tactile response, and/or information previously recorded by user100. Feedback-outputting unit380may comprise appropriate components for outputting acoustical and tactile feedback. For example, feedback-outputting unit380may correspond to hearing interface350and may provide audible feedback to the ear of user100. In some embodiments, processor310may communicate signals with an external feedback outputting unit380via a wireless transceiver360, a wired connection, or some other communication interface. In some embodiments, feedback outputting unit380may also include any suitable display device for visually displaying information to user100.

As further shown inFIG.3, earphone110may include a power source370. The term “power source” includes any device capable of providing electrical power, which can be contained within earphone110or easily carried by hand (e.g., power source370may weigh less than a pound). The mobility of the power source enables user100to use earphone110in a variety of situations. In some embodiments, power source370may include one or more batteries (e.g., nickel-cadmium batteries, nickel-metal hydride batteries, and lithium-ion batteries) or any other type of electrical power supply. In other embodiments, power source370may be rechargeable and contained within the earphone and/or within a casing that holds earphone110. In yet other embodiments, power source370may include one or more energy harvesting devices for converting ambient energy into electrical energy (e.g., portable solar power units, human vibration units, etc.). Power source370may be configured to receive a charge through power interface372. For example, power interface372may include one or more conductive pins or other conductive elements configured to receive energy from an external power source, which may be provided to and retained in power source370.

Earphone110may further include wireless transceiver360, as shown inFIG.3. The term “wireless transceiver” refers to any device configured to exchange transmissions over an air interface by use of radio frequency, infrared frequency, magnetic field, or electric field. Wireless transceiver360may use any known standard to transmit and/or receive data (e.g., Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, or ZigBee). For example, wireless transceiver360may be used for short-range communication with one or more of an additional earphone110, computing device120, earphone case130, or various other devices. In some embodiments, wireless transceiver360may communicate over a network, such as network240. In some embodiments, wireless transceiver360may transmit data (e.g., raw image data, processed image data, extracted information) from earphone110to server250, earphone case130, and/or computing device120. Wireless transceiver360may further be configured to receive data from server250, earphone case130, and/or computing device120. For example, this may include receiving instructions or requests to perform operations (e.g., capture an image, transmit data, etc.), data used for analysis (e.g., models used to process audio signals, voice signature data, facial feature data, etc.), update information (e.g., software updates, firmware updates, etc.), or any other data that may be relevant to processing audio signals using earphone110.

In some embodiments, earphone110may include various controls for adjusting operation of earphone110. For example, this may include one or more buttons, switches, dials, sensors, touch areas, or other components that may trigger or allow for parameters of earphone110to be modified. In some embodiments, earphone110may be equipped with a switch for turning image sensor1410on or off, for entering a battery-saving mode, or the like. Earphone110may include various other controls, for example, for changing an audio sampling rate, an image capture rate, a part of a field of view of image sensor330that is captured, or the like.

As will be appreciated by a person skilled in the art having the benefit of this disclosure, numerous variations and/or modifications may be made to the disclosed embodiments. Not all components are essential for the operation of earphone110. Any component may be located in any appropriate apparatus and the components may be rearranged into a variety of configurations while providing the functionality of the disclosed embodiments. For example, in some embodiments, earphone110may include a camera, a processor, and a wireless transceiver for sending data to another device. Therefore, the foregoing configurations are examples and, regardless of the configurations discussed above, earphone110can capture, store, and/or process images.

FIG.4is a block diagram illustrating the components of an example computing device120. Computing device120may include a processor410, a feedback-outputting unit440, a memory420, a wireless transceiver430, and a display450. Examples of computing device120include a smartphone, a smartwatch, or a tablet having a dedicated application installed therein. In other embodiments, computing device120may include any configuration such as an on-board automobile computing system, a PC, a laptop, and any other system consistent with the disclosed embodiments. In this example, user100may view feedback output in response to identification of vocal trigger, a hand-related trigger on display450or an action performed using computing device120. Additionally, user100may view other data (e.g., images, video clips, object information, schedule information, extracted information, etc.) on display450. In addition, user100may communicate with server250via computing device120.

The term “computing device” refers to a device including a processing unit and having computing capabilities. Some examples of computing device120include a PC, laptop, tablet, or other computing systems such as an on-board computing system of an automobile, for example, each configured to communicate directly with earphone110or server250over network240. Another example of computing device120includes a smartphone having a display450. In some embodiments, computing device120may be a computing system configured particularly for earphone110, and may be provided integral to earphone110or tethered thereto. Earphone110can also connect to computing device120over network240via any known wireless standard (e.g., Wi-Fi, Bluetooth®, etc.), as well as near-filed capacitive coupling, and other short range wireless techniques, or via a wired connection. In an embodiment in which computing device120is a smartphone, a smartwatch, or a tablet, computing device120may have a dedicated application installed therein. For example, user100may view on display450data (e.g., images, video clips, extracted information, feedback information, etc.) that originate from or are triggered by earphone110. In addition, user100may select part of the data for storage in server250.

In some embodiments, processor310may identify environmental information in the captured and analyzed audio and/or images, such as an individual standing in front user100or a known individual speaking, and send computing device120information related to the analyzed information such as the name of the individual and the last time user100met the individual. In a different embodiment, processor410may extract statistical information from captured image data and forward the statistical information to server250. For example, certain information regarding the types of items a user purchases, or the frequency a user patronizes a particular merchant, etc. may be determined by processor410. Based on this information, server250may send computing device120coupons and discounts associated with the user's preferences.

When earphone110is connected or wirelessly connected to computing device120, earphone110may transmit at least part of the image data stored in memory320for storage in memory420. In some embodiments, after computing device120confirms that transferring the part of image data was successful, processor410may delete the part of the image data. The term “delete” means that the image is marked as ‘deleted’ and other image data may be stored instead of it, but does not necessarily mean that the image data was physically removed from the memory.

FIG.5is a block diagram illustrating an example of server250, consistent with embodiments of the present disclosure. As described above, server250may be a computing device and may include one or more dedicated processors and/or memories. For example, server250may include at least one processor, more generally referred to as processor510, a memory (or multiple memories)520, a network interface (or multiple network interfaces)530, as shown inFIG.5. In some embodiments, server250may be a rack of multiple servers. Accordingly, server250may include multiple instances of the example server shown inFIG.5.

Processor510may take the form of, but is not limited to, a microprocessor, embedded processor, or the like, may be integrated in a system on a chip (SoC), or more take the form of any processor described earlier. Furthermore, according to some embodiments, the processor510may be from the family of processors manufactured by Intel®, AMD®, Qualcomm®, Apple®, NVIDIA®, or the like. Processor510may also be based on an ARM architecture, a mobile processor, or a graphics processing unit, etc. The disclosed embodiments are not limited to any type of processor included in server⋅250. In some embodiments, processor510may refer to multiple processors.

Memory520may include one or more storage devices configured to store instructions used by the processor510to perform functions related to the disclosed embodiments. Memory520may be configured to store software instructions, such as programs, that perform one or more operations when executed by the processor510to perform the various functions or methods described herein. The disclosed embodiments are not limited to particular software programs or devices configured to perform dedicated tasks. For example, memory520may store a single program, such as a user-level application, that performs the functions of the disclosed embodiments, or may include multiple software programs. Additionally, the processor510may in some embodiments execute one or more programs (or portions thereof) remotely located from server⋅. Furthermore, the memory520may include one or more storage devices configured to store data for use by the programs. In some embodiments, memory520may include a local database252. In some embodiments, memory520and/or database252may store information associated with user100. For example, server250may maintain a database of acquaintances or contacts of user100, associated images or data used to recognize individuals, audio clips or voiceprints used to recognize individuals, or the like.

Network interface530may include one or more network adaptors or communication devices and/or interfaces (e.g., WiFi®, Bluetooth®, RFID, NFC, RF, infrared, Ethernet, etc.) to communicate with other machines and devices, such as with other components of system200through network240. For example, server250may use a network interface530to receive and transmit information associated with captured and/or processed audio signals within system200.

Earphone and Camera Integration

As described herein, the disclosed embodiments may include selectively conditioning an audio signal to separate speakers, remove background noise or other sounds, and transmit speech or other desired sounds to a hearing interface device. In some embodiments, the various components for capturing information from the environment of the user, analyzing and processing the information, and presenting the processed audio to a user may be contained within a single housing. As people now tend to wear earphones frequently, wearing this single earphone may provide significant functionality as described above, without attracting undesired attention. Further, in some embodiments, a user may wear an earphone in each ear to improve perception of the environment of the user and the processing and presentation of audio signals. For example, multiple earphones may allow for better spatial tracking of audio sources, simultaneous processing functions, or various other techniques described below.

FIGS.6A,6B,6C, and6Dillustrate example configurations of earphone110, consistent with the disclosed embodiments. InFIGS.6A and6B, earphone110is shown inserted into an ear of a user. As shown inFIG.6A, earphone110may include a housing610configured to house or contain the various components of earphone110described above. Housing610may be at least partially insertable into an ear of a user such that hearing interface350may deliver sounds to the user. Housing610may be a rigid or semi-rigid component and may be constructed of any suitable material, such as silicone, rubber, plastic, metal, wood, or the like. In some embodiments, earphone110may include a head part612and a neck part614, as shown inFIG.6A. In this example, earphone110may be used by inserting the head part612(which may include hearing interface350) into the ear, such that neck part614points downward. Image sensor330may be installed in earphone110and, in particular, in the head part614and may face forward such that a line of sight of image sensor330is substantially parallel to the line of sight of a user wearing earphone110.FIG.6Bshows another example embodiment of earphone110, consistent with the disclosed embodiments. In this example, neck part614may point forward, and image sensor330may be installed at the end of neck part614such that a line of sight of image sensor330is again substantially parallel to the line of sight of the user. Thus, in both examples above, earphone110includes a housing610, at least part of which may be insertable into an ear of a user.

FIGS.6C and6Dillustrate another example configuration of hearing interface device, consistent with the disclosed embodiments.FIGS.6C and6Dillustrate earphone110when it is not inserted into the ear of a user.FIG.6Dshows an alternate view of the example configuration of earphone110from the view shown inFIG.6C. In this example configuration, image sensor330may be placed on head part612, similar to the configuration shown inFIG.6A. As shown inFIG.6C, earphone110may include an ear tip620configured to be inserted into an ear of a user. Ear tip620may be a rubberized ear tip configured to provide better fit, additional comfort, and/or isolate background noise when inserted into the ear of the user. In some embodiments an additional microphone may be included in ear tip620, which may be used to improve noise cancelation, determine whether earphone110is properly inserted, or various other applications. In some embodiments, earphone110may further include an acoustic vent608, which may improve the sound quality of earphone110for a user. In some embodiments, earphone110may further include various support components or structures configured to hold earphone110in place when inserted into the ear of a user. For example, this may include an antihelix fastener630. Antihelix fastener630may be a rubberized component configured to fit into the antihelix portion of an ear of a user when earphone110is inserted into the ear of the user to secure earphone110in place. In some embodiments, earphone110may include a sensor606to detect when earphone110is inserted into an ear of the user. For example, sensor606may include an accelerometer, a proximity sensor, a light sensor, a touch sensor, a capacitive or other electrical sensor, an ultrasonic sensor, a microphone, or any other sensor that may indicate earphone110has been inserted into an ear of a user.

In this example, earphone110may include one or more microphones. For example, earphone110may include a microphone340A included on neck part614(shown inFIG.6C) and an additional microphone340B (shown inFIG.6D). Microphones340A and340B may correspond to microphone340described above. The use of multiple microphones340A and340B may allow earphone110to capture sounds from multiple angles, which may ensure that more sounds are captured. In some embodiments, differences in sounds captured using microphones340A and340B may be used to ascertain information about an environment of earphone110. For example, this may include determining a direction of an audio source relative to earphone110as described further herein.

In some embodiments, earphone110may include various other interface components, as shown inFIGS.6C and6D. For example, earphone110may include one or more buttons, dials, switches, touch areas, or pressure-sensitive areas, as described above. In some embodiments, this may include pressure-sensitive area604, which may allow a user to provide an input to earphone110. For example, the user may apply pressure to pressure-sensitive area604, which may indicate that earphone110should change a volume level, capture an image, mute or pause audio playback, change modes (e.g., turning on or off background noise suppression, etc.), or various other commands. Earphone110may further include an indicator602(which may correspond to feedback outputting unit380) to provide status information or other information to a user. For example, indicator602may be a LED or other light source, which may provide information either as a binary indicator (e.g., on and off) or based on a property, such as a color of light, a rate or pattern of blinking, a brightness, or various other properties. Indicator602may indicate whether earphone110has been inserted properly, whether a recognized individual has been detected, a battery level or status, an operating mode, or various other information. In some embodiments, earphone110may further include a charging port640, which may correspond to power interface372described above. For example, charging port640may include one or more pins642or sockets configured to provide electrical contact with a charging device. Accordingly, charging port640may be used to charge a battery of earphone110. Earphone110may include various other ports, such as a data port, which may provide a connection for uploading data to earphone110or downloading data from earphone110. It is to be understood that the example configuration of earphone110shown inFIGS.6A,6B,6C, and6Dare provided by way of example, and earphone110may include various other components.

FIG.6Eillustrates an example use of earphone110for capturing images from an environment of a user100, consistent with the disclosed embodiments. As described above, earphone110may be at least partially inserted into an ear of user100. In this example, earphone110may be configured similar to the configuration illustrated inFIG.6A. Accordingly, image sensor330may be located on a head part of earphone110, as shown. Image sensor330may be configured to capture one or more images from the environment of user100(and earphone110).

In this example, a line of sight of image sensor330may be substantially parallel to a line of sight of user100. By being placed in a hearing interface device inserted into the ear, image sensor330may more accurately follow the head movements of user100. Because image sensor330is placed closer to the eyes of the user as compared to a device worn on the shirt of a user or other placements, a narrower viewing angle θ of image sensor330may be used as compared to a viewing angle that may be required in other placement locations. For example, if an image sensor (e.g., image sensor320) were instead worn on a chest of user100, a relatively large viewing angle may be required to capture representations of objects viewed by user100. In this example, viewing angle θ may range from 50° to 120° (e.g., 70°, 80°, 90°, etc.), whereas an image sensor placed on the chest of user100may require an angle of 130° or greater, depending on the placement. This narrow angle may enable the use of a smaller image size (e.g., a Video Graphics Array (VGA) image rather than larger image sizes of 4000×3000 pixels, etc.). This reduced image size may therefore reduce the memory requirements, processing demands, battery life, or other requirements of earphone110.

In some embodiments, earphone110may be configured to cause an insertion indication to be presented to the ear of user100when earphone110is inserted into the ear of user100. For example, this may be a tone, a chime, or another sound confirming earphone110has been inserted. The insertion of earphone110may be detected in various ways. In some embodiments, earphone110may include an accelerometer which may detect a motion of earphone110. Earphone110may be configured to detect a motion pattern based on an output of the accelerometer, which may be compared to a stored motion pattern or motion characteristics associated with an insertion event. Based on the comparison, the insertion of earphone110may be detected. As another example, the insertion event may be detected based on at least one image captured by image sensor330. For example, processor310may be configured to determine or estimate a height of image sensor330and may determine that the height is consistent with an expected height of image sensor330when placed in the ear of a user. As another example, processor310may recognize an ear, a cheek, a shoulder, or another part of user100within an image, which may indicate earphone110has been inserted.

Similarly, processor210may be configured to determine whether earphone110has been inserted correctly or incorrectly. For example, a correct insertion may be determined based on a direction or orientation of image sensor330. If image sensor330is not properly aligned with a look direction of user100(e.g., within a threshold alignment of a preferred alignment direction), earphone110may be determined to be improperly inserted. In some embodiments, earphone110may be configured to present a different sound when earphone110is improperly inserted. For example, the insertion indication may be a first sound when the housing is correctly inserted into the ear of the user and a second sound when the housing is not correctly inserted into the ear of the user.

In some embodiments, user100may wear multiple earphones110, for example, one in each ear.FIG.7illustrates an example use of multiple hearing interface devices by user100, consistent with the disclosed embodiments. In this example, user100may wear earphone110in a first ear, as described above with respect toFIG.6E. User100may also wear an additional earphone710in a second ear. Additional earphone710may be configured as a mirror image of earphone110such that it fits properly in the other ear of user100. Additional earphone710may include an image sensor730and microphone740, which may be similar to image sensor730and microphone740described above.

Earphone110and additional earphone710may be configured to share information, which may be used to process captured audio signals. For example, wireless transceiver360may be configured to communicate with a corresponding wireless transceiver of additional earphone710. This may include transmitting captured images or audio signals or at least partially processed images or audio signals. In some embodiments earphone110and additional earphone710may share data derived from processing captured audio signals or images. For example, this may include voiceprints, facial features, information identifying a recognized individual, a direction of an audio source, or other information that may be derived from a captured image or audio signal. Accordingly, in some embodiments, processor310may generate processed audio signals based at least in part on information received from additional earphone710. Similarly, a processor of additional earphone710may generate processed audio signals based at least in part on information received from earphone110. For example, each device may perform separate processing and transmit the results to the other device. In some embodiments, the two devices may operate in a hierarchal manner. For example, earphone110may be a primary device and may delegate one or more operations to additional earphone710(or vice versa). Alternatively or additionally, earphone110and additional earphone710may operate autonomously of each other.

This shared information may allow for improved analysis of the environment of user100. For example, as described above, a directionality of a sound may be used in selective conditioning or other forms of processing. This may include amplifying sounds received from a look direction of user100relative to sounds received from other directions, or other forms of direction-based sound processing. By using multiple microphones, each microphone may be used to determine a direction of an audio source, such that a more accurate combined direction of the audio source may be determined. For example, microphones340and740may be used to perform a beamforming technique or similar technique for localizing audio sources. Similarly, image data from image sensors330and730may be used to determine spatial information associated with the environment of user100. For example, the lateral separation between image sensor330and image sensor730may provide parallax information for stereo analysis by processor310(and/or a processing device of additional earphone710). Stereo image analysis may refer to instances where image analysis is performed based on two or more images captured with one or more variations of an image capture parameter. For example, captured images suitable for performing stereo image analysis may include images captured: from two or more different positions, from different fields of view, using different focal lengths, along with parallax information, etc. In some embodiments, this may include sharing data from other sensors, such as accelerometer data or the like.

The sharing of information between earphone110and additional earphone710may also allow the processing demand to be split among multiple processing devices, which may reduce the processing demand on processor310and thus reduce its energy consumption. This sharing may also enable tasks which otherwise cannot be performed due to insufficient processing capabilities that can be fitted into an earphone. Accordingly, various tasks or analyses described herein may be split between earphone110and additional earphone710. For example, this may include splitting processing tasks based on a type of analysis. As one example, earphone110may be configured to process lip movements of individuals or detect facial features, as described above, whereas additional earphone710may be configured to determine voice prints of individuals in the environment of user100(which may further include recognizing one or more individuals within the environment of user100based on stored voice signatures). As another example, tasks may be split by objects or individuals being detected, tracked, and/or recognize. For example, earphone110may be configured to identify a first individual and additional earphone710may be configured to identify a second individual. While various distributions of tasks are provided above, it is to be understood that various other distributions of tasks may be performed. In some embodiments, the shared processing between earphone110and additional earphone710may not be split by discrete tasks and may be distributed in other ways. For example, as noted above, processor310may be a primary processor and may assign specific processing operations to additional earphone710and the result of the processing operations may be provided to processor330.

Selective Conditioning of Audio Signals

In some embodiments, earphone110may be configured to selectively condition audio signals captured from one or more audio sources within the environment of a user.FIG.8illustrates an example environment800of user100in which earphone110may be used for generating processed audio signals, consistent with the disclosed embodiments. As noted above, processor310may be configured to generate one or more processed audio signals based on captured audio signals, captured images, or both. For example, environment800may include individuals810and820, which may be the source of sounds812and822, respectively. In this example, sounds812and814may be voices of individuals810and820. Environment800may further include a sound832, which may be a background or ambient noise from various additional sources. Microphone340may be configured to capture audio signals associated with sounds812,822, and832. Processor310may be configured to selectively condition sounds812,822, and832. For example, selective conditioning may include amplifying audio signals determined to correspond to sound812relative to other audio signals. Amplification may be accomplished digitally, for example by processing audio signals associated with sound812relative to other signals. Amplification may also be accomplished by changing one or more parameters of audio sensor340to focus on audio sounds emanating from region802(e.g., a region of interest) associated with user look direction804. For example, audio sensor340may be a directional microphone and processor310may perform an operation to focus audio sensor340on sound812or other sounds within region802. Various other techniques for amplifying sound812may be used, such as using a beamforming microphone array, acoustic telescope techniques, etc.

In some embodiments, selective conditioning may also include attenuation or suppressing one or more audio signals received from directions outside of region802. For example, processor812may attenuate sounds822and832. Similar to amplification of sound812, attenuation of sounds may occur through processing audio signals, or by varying one or more parameters associated with one or more microphones820to direct focus away from sounds emanating from outside of region802.

In some embodiments, conditioning may further include changing a tone of audio signals corresponding to sound812to make sound812more perceptible to user100. For example, user100may have lesser sensitivity to tones in a certain range and conditioning of the audio signals may adjust the pitch of sound812to make it more perceptible to user100. For example, user100may experience hearing loss in frequencies above 10 khz. Accordingly, processor310may remap higher frequencies (e.g., at 15 khz) to 10 khz.

In some embodiments processor310may be configured to change a rate of speech associated with one or more audio signals. Accordingly, processor310may be configured to detect speech within one or more audio signals received by audio sensor340, for example using voice activity detection (VAD) algorithms or techniques. If sound812is determined to correspond to voice or speech, for example from individual810, processor310may be configured to vary the playback rate of sound812. For example, the rate of speech of individual810may be decreased to make the detected speech more perceptible to user100. Various other processing may be performed, such as modifying the tone of sound812to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal. If speech recognition has been performed on the audio signal associated with sound812, conditioning may further include modifying the audio signal based on the detected speech. For example, processor310may introduce pauses or increase the duration of pauses between words and/or sentences, which may make the speech easier to understand.

The conditioned audio signal may then be transmitted to hearing interface350and produced for user100. Thus, in the conditioned audio signal, sound812may be easier to hear to user100, louder and/or more easily distinguishable than sounds822and832, which may represent background noise within the environment.

Various techniques may be used for identifying which sounds should be amplified or attenuated relative to other sounds. For example, this may include determining a look direction804of user100. Look direction804may be determined based on a field of view802of image sensor820. Accordingly, because individual810is in the look direction of user100, sound812may be amplified relative to one or both of sounds822and832. In some embodiments, sounds may be amplified or identified based on an identification of an individual in environment800, as described below.

In embodiments where user100is wearing an additional earphone710, information from microphone740and/or image sensor730may be used to improve the analysis of environment800. For example, a direction associated with sound812(or various other sounds in environment800) may be determined based on a combined analysis of audio signals captured from microphones340and740(e.g., through beamforming or other techniques). In some embodiments, a direction associated with sound812may be determined separately based on microphones340and740and these directions may be combined (e.g., averaged together, etc.) to determine a more accurate direction of individual810relative to user100. For example, each of microphones340and740may be a directional microphone capable of determining a directionality of sound812. Alternatively or additionally, a direction of sound812may be determined based on a combined analysis of audio signals captured by microphones340and740. For example, processor310(and/or a processor of additional earphone710) may analyze differences in amplitude or other characteristics (e.g., timing, etc.) of sound812as it is represented in the respective captured audio signals. If sound812is represented with a slightly higher amplitude in earphone110than in additional earphone710, this may indicate that sound812is being received from a direction slightly to the right of user100, as shown inFIG.8. Similarly, sound832may be significantly louder or clearer as captured by earphone110relative to additional earphone710, which may indicate it is coming from the right of user100. Similarly, images captured by image sensors330and730may be processed using a stereo image analysis, which may allow processor310to determine a depth and/or position of individuals810and820relative to user100.

In some embodiments, earphone110and additional earphone710may be configured to synchronize with each other so that the timing at which the processed audio signal is presented to the user in each ear is consistent. For example, it may be disorienting or unpleasant to the user if a slight delay exists between the audio presented in each ear. In some embodiments, the timing between earphone110and additional earphone710may be synchronized based on a timing of sounds captured by microphones340and740. As another example, the timing between earphone110and additional earphone710may be synchronized based on a timing of visual events captured by image sensors330and730, such as an individual opening his mouth, clapping his hands or the like. This may provide a more accurate timing synchronization as light will be received at substantially the same time by both devices, whereas a slight delay may exist for captured audio due to differences between the speed of light and the speed of sound.

Selective Conditioning Based on Recognition of Individuals

In some embodiments, earphone110may selectively amplify audio signals associated with a voice of a recognized individual. The hearing aid system may store voice characteristics and/or facial features of a recognized person to aid in recognition and selective amplification. For example, when an individual enters the field of view of earphone110, the individual may be recognized as an individual that has been introduced to the device, or that has possibly interacted with user100in the past (e.g., a friend, colleague, relative, prior acquaintance, etc.). Accordingly, audio signals associated with the recognized individual's voice may be isolated and/or selectively amplified relative to other sounds in the environment of the user. Audio signals associated with sounds received from directions other than the individual's direction may be suppressed, attenuated, filtered or the like.

Accordingly, earphone110may be configured to recognize an individual in the environment of user100.FIG.9Ais a schematic illustration showing an exemplary environment900for selective conditioning of audio signals based on recognition of an individual, consistent with the present disclosure. Earphone110may be configured to recognize a face911or voice912associated with an individual910within the environment of user100. For example, earphone110may be configured to capture one or more images of the surrounding environment of user100using image sensor330. The captured images may include a representation of a recognized individual910, which may be a friend, colleague, relative, or prior acquaintance of user100. Processor210may be configured to analyze the captured images and detect the recognized user using various facial recognition techniques, as represented by element911. Accordingly, earphone110, or specifically memory320, may comprise one or more facial or voice recognition components.

Accordingly, earphone110may be configured to identify one or more faces within the environment of user100. For example, earphone110may identify facial features on the face911of individual910, such as the eyes, nose, cheekbones, jaw, or other features. Earphone110(e.g., via processor310) may then analyze the relative size and position of these features to identify the user. Earphone110may utilize one or more algorithms for analyzing the detected features, such as principal component analysis (e.g., using eigenfaces), linear discriminant analysis, elastic bunch graph matching (e.g., using Fisherface), Local Binary Patterns Histograms (LBPH), Scale Invariant Feature Transform (SIFT), Speed Up Robust Features (SURF), or the like. Other facial recognition techniques such as 3-Dimensional recognition, skin texture analysis, and/or thermal imaging may also be used to identify individuals. Other features besides facial features may also be used for identification, such as the height, body shape, or other distinguishing features of individual910.

Earphone110may access a database or data associated with user100to determine if the detected facial features correspond to a recognized individual. For example, a processor310may access a database (e.g., database252) containing information about individuals known to user100and data representing associated facial features or other identifying features. Such data may include one or more images of the individuals, or data representative of a face of the user that may be used for identification through facial recognition. Database252may be any device capable of storing information about one or more individuals, and may include a hard drive, a solid state drive, a web storage platform, a remote server, or the like. Database252may be located within earphone110(e.g., within memory320) or external to earphone110(e.g., at case130, server250as shown inFIG.2). In some embodiments, database252may be associated with a social network platform, such as Facebook™, LinkedIn™, Instagram™, etc. Earphone110may also access a contact list of user100, such as a contact list on the user's phone, a web-based contact list (e.g., through Outlook™, Skype™, Google™, SalesForce™, etc.) or a dedicated contact list associated with hearing interface device810. In some embodiments, database252may be compiled by earphone110through previous facial recognition analysis. For example, processor310may be configured to store data associated with one or more faces recognized in images captured by earphone110in database252. Each time a face is detected in the images, the detected facial features or other data may be compared to previously identified faces in database252. Earphone110may determine that an individual is a recognized individual of user100if the individual has previously been recognized by the system in a number of instances exceeding a certain threshold, if the individual has been explicitly introduced to earphone110, or the like.

In some embodiments, processor310may use various techniques to recognize the voice of individual910, as described in further detail below. The recognized voice pattern and the detected facial features may be used, either alone or in combination, to determine that individual910is recognized by earphone110. Processor310may also determine a user look direction804, as described above, which may be used to verify the identity of individual910. For example, if user100is looking in the direction of individual910(especially for a prolonged period), this may indicate that individual910is recognized by user100, which may be used to increase the confidence of earphone110or other identification means.

Processor310may further be configured to determine whether individual910is recognized by user100based on one or more detected audio characteristics of sounds associated with a voice of individual910. Returning toFIG.9A, processor310may determine that sound920corresponds to voice912of individual910. Processor310may analyze audio signals representative of sound920captured by audio sensor340(which may correspond to audio sensor560) to determine whether individual910is recognized by user100. This may be performed using one or more voice recognition algorithms, such as Hidden Markov Models, Dynamic Time Warping, neural networks, or other techniques. Processor310may access database252, which may further include a voiceprint of one or more individuals. Processor310may analyze the audio signal representative of sound920to determine whether voice912matches a voiceprint of an individual in database252. Accordingly, database252may contain voiceprint data associated with a number of individuals, similar to the stored facial identification data described above. After determining a match, individual910may be determined to be a recognized individual of user100. This process may be used alone, or in conjunction with the facial recognition techniques described above. For example, individual910may be recognized using image-based recognition and may be verified using voice recognition, or vice versa.

In some embodiments, an individual may be recognized at least in part using an input from user100. For example, user100may provide a user input through computing device120, which may be used to select and/or identify an individual (or confirm an identity of the individual) alone or on combination with the voice- or image-based recognition described above.FIG.9Billustrates an example user interface950that may be used to receive an input identifying an individual, consistent with the disclosed embodiments. In some embodiments, user interface950may be displayed on a device accessible to user100, such display450of computing device120. In some embodiments, user interface950may be displayed automatically. For example, system200may detect that an individual is speaking and may prompt user100to identify the individual through user interface950. In some embodiments, this may be based on a determination that the individual is unrecognized or that the individual is recognized with a low confidence. For example, system200may attempt to recognize individual910based on face911or voice912, as described above. If a match is not identified, or if a match is determined to have a low confidence (e.g., by determining a confidence score based on a degree of matching and comparing the confidence score to a confidence threshold), system200may prompt user100to provide additional input through user interface950. Alternatively or additionally, user interface950may be accessed by user100, for example, by selecting an option to manually identify an individual.

As shown inFIG.9B, user interface950may include one or more elements954representing individuals that may correspond to individual910. In some embodiments, user interface950may display individuals known by or associated with user100. For example, system200may access a database (e.g., database252) containing information about individuals known to user100, such as friends, colleagues, acquaintances, or other contacts. Accordingly, elements954may allow user100to easily and quickly select an individual to be associated with an individual present in the current environment of user100. In some embodiments, elements954may be sorted according to various criteria. For example, if a voice or face of individual910is detected, elements954may be sorted according to a best match to the detected voice or facial features (e.g., based on a confidence score indicating a degree of match to a stored voiceprint or facial feature data). Alternatively or additionally, elements954may be sorted based on other factors, such as a frequency at which the individual is encountered by user100, a closeness of relationship to user100, or similar factors. In some embodiments, user interface950may include other elements for specifying an individual. For example, user interface950may include a text-based search element952for searching for an individual based on a name or other characteristic of the individual. As another example, user interface950may include an element956to create a new contact, which may open a new interface to allow user100to specify the name of an individual and/or other information. Once an individual has been selected or otherwise specified through user interface950, system200may update a profile associated with the specified individual. For example, if user100selects element954, system200may update stored profile information based on face911, voice912, or any other relevant information.

In some embodiments, earphone110may detect the voice of an individual that is not within the field of view of earphone110. Returning toFIG.9A, this may include determining that one or more of sounds921and921, which may correspond to the voice of an individual, are coming from outside of region930. For example, the voice may be heard over a speakerphone, from a back seat, or the like. In such embodiments, recognition of an individual may be based on the voice of the individual only, in the absence of a speaker in the field of view or in the absence of an image sensor. Processor110may analyze the voice of the individual as described above, for example, by determining whether the detected voice matches a voiceprint of an individual in database252.

After determining that individual910is a recognized individual of user100, processor310may cause selective conditioning of audio associated with the recognized individual. The conditioned audio signal may be transmitted to hearing interface device810, and thus may provide user100with audio conditioned based on the recognized individual. For example, the conditioning may include amplifying audio signals determined to correspond to sound920(which may correspond to voice912of individual910) relative to other audio signals. In some embodiments, amplification may be accomplished digitally, for example by processing audio signals associated with sound920relative to other signals. Additionally, or alternatively, amplification may be accomplished by changing one or more parameters of audio sensor340to focus on audio sounds associated with individual910. For example, audio sensor340may be a directional microphone and processor310may perform an operation to focus audio sensor340on sound920. Various other techniques for amplifying sound920may be used, such as using a beamforming microphone array, acoustic telescope techniques, etc.

In some embodiments, processor310may perform further analysis based on captured images or videos to determine how to selectively condition audio signals associated with a recognized individual. In some embodiments, processor310may analyze the captured images to selectively condition audio associated with one individual relative to others. For example, processor310may determine the direction of a recognized individual relative to the user based on the images and may determine how to selectively condition audio signals associated with the individual based on the direction. If the recognized individual is standing to the front of the user, audio associated with that user may be amplified (or otherwise selectively conditioned) relative to audio associated with an individual standing to the side of the user. Similarly, processor310may selectively condition audio signals associated with an individual based on proximity to the user. Processor310may determine a distance from the user to each individual based on captured images or based on voice volume and may selectively condition audio signals associated with the individuals based on the distance. For example, an individual closer to the user may be prioritized higher than an individual that is farther away.

In some embodiments, selective conditioning of audio signals associated with a recognized individual may be based on the identities of individuals within the environment of the user. For example, where multiple individuals are detected in the images, processor310may use one or more facial recognition techniques to identify the individuals, as described above. Audio signals associated with individuals that are known to user100, for example recognized in an image or identified in accordance with one or more voiceprints, may be selectively amplified or otherwise conditioned to have priority over unknown individuals. For example, processor310may be configured to attenuate or silence audio signals associated with bystanders in the user's environment, such as a noisy office mate, etc. In some embodiments, processor310may also determine a hierarchy of individuals and give priority based on the relative status of the individuals. This hierarchy may be based on the individual's position within a family or an organization (e.g., a company, sports team, club, etc.) relative to the user. For example, the user's boss may be ranked higher than a co-worker or a member of the maintenance staff and thus may have priority in the selective conditioning process. In some embodiments, the hierarchy may be determined based on a list or database. Individuals recognized by the system may be ranked individually or grouped into tiers of priority. This database may be maintained specifically for this purpose or may be accessed externally. For example, the database may be associated with a social network of the user (e.g., Facebook™, LinkedIn™, etc.) and individuals may be prioritized based on their grouping or relationship with the user. Individuals identified as “close friends” or family, for example, may be prioritized over acquaintances of the user.

Selective conditioning may be based on a determined behavior of one or more individuals determined based on the captured images. In some embodiments, processor310may be configured to determine a look direction of the individuals in the images. Accordingly, the selective conditioning may be based on behavior of the other individuals towards the recognized individual. For example, processor310may selectively condition audio associated with a first individual that one or more other users are looking at. If the attention of the individuals shifts to a second individual, processor310may then switch to selectively condition audio associated with the second user. In some embodiments, processor310may be configured to selectively condition audio based on whether a recognized individual is speaking to the user or to another individual. For example, when the recognized individual is speaking to the user, the selective conditioning may include amplifying an audio signal associated with the recognized individual relative to other audio signals received from directions outside a region associated with the recognized individual. When the recognized individual is speaking to another individual, the selective conditioning may include attenuating the audio signal relative to other audio signals received from directions outside the region associated with the recognized individual.

In some embodiments, processor310may have access to one or more voiceprints of individuals, which may facilitate selective conditioning of voice912of individual910in relation to other sounds or voices. Having a speaker's voiceprint, and a high quality voiceprint in particular, may provide for fast and efficient speaker separation. A high quality voice print may be collected, for example, when the user speaks alone, preferably in a quiet environment. By having a voiceprint of one or more speakers, it is possible to separate an ongoing voice signal almost in real time, e.g. with a minimal delay, using a sliding time window. The delay may be, for example 10 ms, 20 ms, 30 ms, 50 ms, 100 ms, or the like. Different time windows may be selected, depending on the quality of the voice print, on the quality of the captured audio, the difference in characteristics between the speaker and other speaker(s), the available processing resources, the required separation quality, or the like. In some embodiments, a voice print may be extracted from a segment of a conversation in which an individual speaks alone, and then used for separating the individual's voice later in the conversation, whether the individual's is recognized or not.

In some embodiments, for example when the device does not comprise an image sensor, the user may indicate which individuals he or she would like to hear in the current context. The user may use an I/O device, such as a mobile phone or other device paired to the system, and may select from a list one or more individuals for which the user has a voiceprint.FIG.10illustrates an example user interface1000, which may be used to select a voice or other audio source for selective conditioning, consistent with the disclosed embodiments. User interface1000may include one or more selectable elements1002,1004,1006, and1008, which may represent audio sources in the current environment of user100. In some embodiments, system200may recognize a voice of one or more individuals in the environment of user100. For example, elements1002and1004may be associated with individuals for which a voiceprint is stored. Accordingly, elements1002and1004may include stored information associated with the recognized individual, such as a name of the individual or other personal information. Although elements1002,1004,1006, and1008show images of the individuals, it is to be appreciated that these images may be absent. In some embodiments, a default image may be displayed.

In some embodiments, user interface1000may display additional elements associated with unrecognized individuals. For example, earphone110may detect a voice of a speaker for which no voiceprint match is determined (or a match confidence is below a threshold). Accordingly, user interface1000may display an element1006associated with the user. In some embodiments, user interface1000may include an element1008representing background or ambient noise (e.g., sound922), which may encompass sounds not associated with elements1002,1004, and1006.

User interface1000may allow user100to select one or more audio sources for selective conditioning via elements1002,1004,1006, and1008. Then, when any of the selected individuals speaks, the voice will be amplified or otherwise enhanced. For example, in a restaurant, the user may select to hear the people he is dining with but not others. In the example shown inFIG.10, by selecting element1002, a voice of an individual associated with element1002may be amplified relative to sounds associated with elements1004,1006, and1008. This may include various other forms of selective conditioning as described herein. In some embodiments, user interface1000may allow a user to select multiple elements. For example, if user100selects both of elements1002and1004, voices associated with these sources may be amplified or otherwise conditioned relative to sounds associated with elements1006and1008. In some embodiments, the selective conditioning may be tuned or adjusted for each source. For example, selecting one of elements1002,1004,1006, and1008may bring up a display of one or more options associated with the audio source. For example, this may include selecting and holding an element in user interface1000or various other selection mechanisms. The options may include an adjustable volume control to vary the degree of amplification relative to other audio sources. In some embodiments, the options may also include muting or silencing a particular audio source.

User interface1000may further include an element1010to register a new voiceprint of a user. For example, if the user wishes to add another individual for which no voiceprint is available, the user may select element1010and ask the individual to speak. Computing device120device may be configured to capture the individual's voice and/or to extract a voiceprint and store it in association with an identifier of the individual, as described above. In some embodiments, computing device120may prompt user100to enter additional information about the individual, such as a name, relationship, or other information that may be stored in association with the individual. In some embodiments, system200may store voiceprints for individuals automatically. For example, if an individual associated with element1006is detected but is not recognized, system200may store the voiceprint in association with an identifier of the individual. By selecting element1006, user interface1000may allow user100to add additional information to be stored in association with the individual. In some embodiments, these automatically generated profiles may be temporary. For example, system200may remove a stored voiceprint associated with element1006after a predetermined time interval where the voiceprint is not saved by user100, is not recognized again, or the like.

According to some embodiments, system200may store default preferences associated with one or more individuals. For example, user100may specify that a voice associated with element1002should be amplified or otherwise selectively conditioned by default. Accordingly, user interface1000may initially show element1002as being selected, which user100can manually unselect. In some embodiments, user100may prioritize one or more individuals relative to each other. For example, an individual associated with element1004may have a rank or preference level that is higher than an individual associated with element1002. Accordingly, the voice of an individual associated with element1004may be amplified relative to the voice of an individual associated with element1002by default.

Separating voices may be performed as follows: spectral features, also referred to as spectral attributes, spectral envelope, or spectrogram may be extracted from a clean audio of a single speaker and fed into a pre-trained first neural network, which generates or updates a signature of the speaker's voice based on the extracted features. The audio may be for example, of one second of clean voice. The output signature may be a vector representing the speaker's voice, such that the distance between the vector and another vector extracted from the voice of the same speaker is typically smaller than the distance between the vector and a vector extracted from the voice of another speaker. The speaker's model may be pre-generated from a captured audio. Alternatively or additionally, the model may be generated after a segment of the audio in which only the speaker speaks, followed by another segment in which the speaker and another speaker (or background noise) is heard, and which it is required to separate.

Then, to separate the speaker's voice from additional speakers or background noise in a noisy audio, a second pre-trained neural network may receive the noisy audio and the speaker's signature, and output an audio (which may also be represented as attributes) of the voice of the speaker as extracted from the noisy audio, separated from the other speech or background noise. It will be appreciated that the same or additional neural networks may be used to separate the voices of multiple speakers. For example, if there are two possible speakers, two neural networks may be activated, each with models of the same noisy output and one of the two speakers. Alternatively, a neural network may receive voice signatures of two or more speakers, and output the voice of each of the speakers separately. Accordingly, the system may generate two or more different audio outputs, each comprising the speech of the respective speaker. In some embodiments, if separation is impossible, the input voice may only be cleaned from background noise.

In some embodiments, when an individual has been recognized, earphone110and/or additional earphone710may be configured to present the name of the individual to the ear of the user through hearing interface350. For example, this may include performing a lookup function based on detected facial features and/or voice signatures to determine the name of individual910. In some embodiments, earphone110and additional earphone710may present the name of individual910in a manner that the name will be perceived to the user as coming from the direction of individual910. For example, based on variations in the timing and amplitude of how the name is presented in earphone110and additional earphone710, a direction that the name of the user is presented from may be simulated. This simulated directionality may help user100determine which individual in environment900the name belongs to. Any other sounds presented to user100described herein, such as processed audio signals, etc. may similarly be presented with a simulated directionality when additional earphone710is used.

FIG.11is a flowchart showing an example process1100for generating processed audio signals, consistent with the disclosed embodiments. Process1100may be performed by at least one processing device of a hearing interface device, such as processor310. It is to be understood that throughout the present disclosure, the term “processor” is used as a shorthand for “at least one processor.” In other words, a processor may include one or more structures that perform logic operations whether such structures are collocated, connected, or dispersed. In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process1100. Further, process1100is not necessarily limited to the steps shown inFIG.11, and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process1100.

In step1110, process1100may include receiving a captured audio signal representative of sounds captured from an environment of the hearing interface device. The sounds may be captured by at least one microphone associated with the housing. For example, step1110may include receiving sounds captured by microphone340, as described above. The microphone may be associated with the housing in that it may be at least partially contained within the housing, affixed to the housing, protruding from the housing, or otherwise integrated with the housing.

In step1120, process1100may include receiving at least one image captured from the environment of the hearing interface device. The at least one image may be captured by at least one camera associated with the housing. For example, step1120may include receiving at least one image captured by image sensor330, as described above. The camera may be associated with the housing in that it may be at least partially contained within the housing, affixed to the housing, protruding from the housing, or otherwise integrated with the housing. In some embodiments, the at least one camera may have a line of sight that is substantially aligned to a line of sight of a user of the hearing interface device when the housing is placed in the ear of the user. In some embodiments, the camera may have a relatively narrow view angle. For example, the at least one camera may have an angle of view between 50 degrees to 120 degrees, as described above.

In step1130, process1100may include generating a processed audio signal based on analysis of at least one of the captured audio signal or the at least one image. This may include various forms of processing described throughout the present disclosure. In some embodiments, generating the processed audio signal may include amplifying a voice of an individual represented in the captured audio signal relative to at least one additional sound. As described herein, amplifying the voice of the individual may include separating the voice of the individual from the at least one additional sound. For example, the at least one additional sound may include a background noise and amplifying the voice of the individual relative to the at least one additional sound may include attenuating the background noise.

In some embodiments, the voice of the individual may be amplified based on a determination that the individual is a recognized individual. For example, amplifying the voice of the individual may include identifying the individual based on the at least one image based on facial features, which may be compared to a database of stored facial features associated with known individuals. As another example, amplifying the voice of the individual may include identifying the individual based on a voice signature associated with the individual. This may include determining a voiceprint of the individual and comparing the voiceprint to a database of voice signatures of known individuals. In some embodiments, the voice of the individual may be amplified based on a determination that the individual is speaking. For example, amplifying the voice of the individual may include tracking a lip movement of the individual represented in the at least one image and determining the individual is speaking based on the tracked lip movement. Alternatively or additionally, the voice of the individual may be amplified based on a selection of the individual by user100. For example, computing device120may display user interface1000as described above, which may allow user100to select the individual from a plurality of audio sources detected in the environment of user100. In some embodiments, this ability to selectively condition audio based on selection of an audio source may occur in the absence of image data (e.g., where a camera is blocked or where earphone110does not include a camera).

In some embodiments, the audio signal may be processed based on information received from another hearing interface device, such as additional earphone710described above. Accordingly, process1100may further include receiving data obtained from an associated hearing interface device, which may be configured to be at least partially inserted into an additional ear of the user or placed anywhere else, for example, as a necklace, a pin, attached to the user's glasses or the like, and the processed audio signal may further be generated based on analysis of the received data. In some embodiments, the received data may include at least one of a voice signature, a lip signature, or a facial feature of an individual in the environment of the user. The voice signature, the lip signature, or the facial feature may be determined by the associated hearing interface device based on at least one of the additional captured audio signal or the additional image. In some embodiments, process1100may further include determining at least one of an additional facial feature, an additional lip signature, or an additional voice signature of an additional individual based on at least one of the captured audio signal or the at least one image, the additional individual being different from the individual. In some embodiments, generating the processed audio signal may include determining a direction of at least one audio source represented in the captured audio signal. When an additional hearing interface device is used, this may be based on information captured by both of the hearing interface devices. For example, the direction being determined based on a first indication of direction of the audio source determined based on the captured audio signal and a second indication of direction of the audio source included in the received data.

In step1140, process1100may include causing at least a portion of the processed audio signal to be presented to the ear of the user. For example, this may include presenting the processed audio signal or a portion thereof to an ear of user100using hearing interface350.

In some embodiments, process1100may include additional steps, including those described above. For example, process1100may include causing an insertion indication to be presented to the ear of the user when at least a portion of the housing is inserted into the ear of the user. As described above, this may further include determining whether the housing is correctly inserted into the ear of the user. The insertion indication may comprise a first sound when the housing is correctly inserted into the ear of the user and a second sound when the housing is not correctly inserted into the ear of the user. The determination whether the housing is correctly inserted into the ear of the user may be based on an indication received from an accelerometer included in the housing, a sensor output (e.g., an output of sensor606), an image captured using image sensor330or any other information that may indicate a proper insertion.

In some embodiments, process1100may further include providing information for locating the hearing interface device to a user. For example, process1100may include receiving, from an external source, a request to provide information for locating the hearing interface device. The external source may be a server (e.g., server250), an auxiliary device (e.g., computing device120), an additional hearing interface device (e.g., additional earphone710), or various other sources. Process1100may include causing, based on the request, locating information to be captured from the environment of the hearing interface device based on the received request. The locating information may include at least one of a locating image captured using the at least one camera (e.g., image1020) or a locating audio signal obtained using the at least one microphone. Process1100may then include transmitting the locating information to a predetermined destination. For example, this may include transmitting the locating information to a server (e.g., server250), an auxiliary device (e.g., computing device120), an additional hearing interface device (e.g., additional earphone710), or various other destinations. In some embodiments, the predetermined location may be associated with the external source (e.g., a server). For example, if the request is received from the server, the locating information may be transmitted back to the server (i.e., a predetermined location at or associated with the server). Alternatively or additionally, the predetermined location may not be associated with the external source.

As another example, process1100may further include generating an indication of a detected condition of the hearing interface device. For example, process1100may include detecting a condition associated with the hearing interface device and causing a status indication to be presented to the ear of the user based on the detected condition, as described above. In some embodiments, the condition may include an obstruction of a field of view of the at least one camera, which may be detected based on analysis of the at least one image (e.g., based on analysis of image1010as described above), a battery health status, or the like.

Earphone Smartcase for Audio Processing

As described herein, system200may include an earphone case130, which may be used to store and/or charge one or more earphones110. For example, earphone case130may include one or more compartments for receiving earphones110and may include one or more power interfaces for replenishing a charge for power source370. Earphone case130may further include a substantially rigid outer housing to protect earphones110from damage or exposure to foreign substances when not in use. In some embodiments, earphone case130may further include one or more processors for processing audio signals captured by earphone110. Thus, the processing within system200may be distributed between processors located within the housings of the one or more earphones110, and one or more processors located within earphone case130.

In some embodiments, processing tasks may be distributed based on a processing demand or other properties associated with the processing task. For example, processing tasks that are computationally inexpensive, that do not require application of a trained machine learning model, or other categories of tasks may be performed by the processors within the earphone housing. For example, processing performed at earphone110may include filtering stationary noise, equalizing, volume control, communication with an external device such as a smartphone, a laptop, or the like. Further processing, which may be more resource consuming, may be performed by a stronger processor, located within the storage and charging earphone case130. For example, processing performed at earphone case130may include spectral subtraction, providing communication between the two earphones, speech enhancement or any other forms of processing tasks. Accordingly, the inclusion of a processing device in earphone case130may allow for more efficient processing of audio signals through distributed processing within system200. The disclosed embodiments may further reduce the processing requirements for processor310and thus may allow for a more compact or cost-effective design of earphone110.

FIG.12is a block diagram illustrating the components of earphone case130according to an example embodiment. As shown inFIG.12, earphone case130may include a processor1210, an audio sensor1240, an image sensor1230, a wireless transceiver1260, a memory1220, a power source1270, one or more compartments1202and1204, and one or more power interfaces1272,1274, and1276. Earphone case130is not limited to the components shown inFIG.12and may include other of fewer components depending on the application. For example, image sensor1230may not be present in earphone case130in some embodiments. For example, in some embodiments earphone case130may also include buttons, other sensors such as a proximity sensor or similar sensors, and inertial measurements devices such as accelerometers, gyroscopes, magnetometers, temperature sensors, color sensors, light sensors, etc. Earphone case130may further include a data port with one or more suitable interfaces for connecting with an external device (not shown). In some embodiments, earphone case130may not include an image sensor and/or an audio sensor.

As with processor310, processor1210may include any suitable processing device and may include more than one processing device. For example, processor1210may include one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logic operations. The instructions executed by processor1210may, for example, be pre-loaded into a memory integrated with or embedded into processor1210or may be stored in a separate memory (e.g., memory1220). Processor1210may be configured to generate a processed audio signal based on analysis of a captured audio signal or a captured image, as described throughout the present disclosure. For example, this may include executing instructions stored in memory1220to perform selective conditioning of audio signals, analyze lip movements or other gestures and/or voice signatures, or various other analyses described above with respect to processor310. Processor310may process audio signals (or portions thereof) captured using audio sensor340of earphone110and received through wireless transceiver1260. In some embodiments, this may further include analyzing image data (which may be captured using image sensor330,1230, or both) and/or audio data captured using audio sensor1240. Memory1220may comprise a Random Access Memory (RAM), a Read-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium, a flash memory, other permanent, fixed, or volatile memory, or any other mechanism capable of storing instructions.

Image sensor1230, if present, may be configured to detect and convert optical signals into electrical signals, which may be used to provide information for an image or a video stream (i.e., image data) based on the detected signal, similar to image sensor320. For example, image sensor1230may include a semiconductor charge-coupled device (CCD), an active pixel sensor in complementary metal-oxide-semiconductor (CMOS), or an N-type metal-oxide-semiconductor (NMOS or Live MOS). Image sensor1230may be part of a camera included in the same housing as earphone case130. Although image sensor1230is described generally herein as an optical camera, it is to be understood that image sensor1230is not limited to this form of capturing device, and any sensor that outputs information about the structure or appearance of the environment may be used, such a video camera, a still camera, a wide angle camera, a narrow angle camera, a thermal camera, a radar device, a Simultaneous Localization and Mapping (SLAM) device, or various other forms of sensors. In some embodiments, the camera may be a black and white camera, which may have advantages such as a better signal-to-noise (SNR) ratio, a better resolution, a simpler lens or other components, or the like.

Audio sensor1240may be configured to capture one or more sounds from an environment of earphone case130, similar to audio sensor340described herein. In some embodiments, audio sensor1240may also be configured to determine a directionality of sounds in the environment of a user. For example, audio sensor1240may comprise one or more directional microphones, which may be more sensitive to picking up sounds in certain directions. In some embodiments, audio sensor1240may comprise a unidirectional microphone, designed to pick up sound from a single direction or small range of directions. Audio sensor1240may also comprise a cardioid microphone, which may be sensitive to sounds from the front and sides. Audio sensor1240may also include a microphone array, and thus audio sensor1240may include multiple microphones.

Earphone case130may further include a wireless transceiver1260, which may be the same as or similar to wireless transceiver360described above. Wireless transceiver1260may use any known standard to transmit and/or receive data (e.g., Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, or ZigBee). In some embodiments, wireless transceiver1260may communicate over a network (e.g., network240), which may include the Internet, a wired Wide Area Network (WAN), a wired Local Area Network (LAN), a wireless WAN (e.g., WiMAX), a wireless LAN (e.g., IEEE 802.11, etc.), a mesh network, a mobile/cellular network, an enterprise or private data network, a storage area network, a virtual private network using a public network, or various other types of network communications. In some embodiments, wireless transceiver1260may transmit and receive data through communications with one or more of earphone(s)110, computing device120, and/or server250, as indicated inFIG.2. For example, this may include receiving instructions or requests to perform operations (e.g., capture an image, transmit data, generate a processed audio signal, etc.), data used for analysis (e.g., models or other programming tools used to process audio signals, voice signature data, facial feature data, etc.), update information (e.g., software updates, firmware updates, etc.), or any other data that may be relevant to processing audio signals. Although wireless transceiver1260is shown to communicate directly with various components within system200, in some embodiments, these communication paths may include various intermediate components.

In some embodiments, processor1210may communicate data to feedback-outputting unit1280, which may include any device or component configured to provide information to a user100. Feedback outputting unit1280may include one or more feedback systems for providing the output of information to user100. In the disclosed embodiments, the audible or visual feedback may be provided via any type of connected audible, visual or tactile system or any combination of the above. Feedback of information according to the disclosed embodiments may include audible feedback to user100(e.g., using a Bluetooth™ or other wired or wirelessly connected speaker, or a bone conduction headphone). Feedback outputting unit1280of some embodiments may additionally or alternatively produce a visible output of information to user100, for example, through an LED, display, or other visual indicator provided as part of earphone case130(or a separate device, such as computing device120). Feedback outputting unit1280may be provided as part of earphone case130(as shown) or may be provided external to earphone case130and communicatively coupled thereto, for example as part of earphone110.

As with power source370described above, power source1270may include one or more batteries (e.g., nickel-cadmium batteries, nickel-metal hydride batteries, and lithium-ion batteries) or any other type of electrical power supply. In other embodiments, power source1270may be rechargeable and contained within the housing of earphone case130. In yet other embodiments, power source1270may include one or more energy harvesting devices for converting ambient energy into electrical energy (e.g., portable solar power units, human vibration units, etc.). Power interface1276may include one or more conductive pins, ports, cables, or other conductive elements configured to receive energy from an external power source, which may be provided to and retained in power source1270.

As further shown inFIG.12, earphone case130may include at least one compartment, such as compartments1202and1204. Compartments1202and1204may each be configured to receive an earphone110. For example, compartment1202may be configured to receive earphone110and compartment1204may be configured to receive earphone710. Accordingly, earphone case130may be configured to store one or more earphones within compartments1202and1204. While two separate compartments are shown inFIG.12by way of illustration, various other configurations may be used. For example, a single compartment may be configured to hold multiple earphones110.

Earphone case130may further include one or more power interfaces configured to provide power to earphone110. For example, earphone case130may include power interface1272and power interface1274, as shown. Power interfaces1272and1274may each be configured to electronically couple with power interface372of a corresponding earphone110when earphone110is housed in earphone case130. Through this coupling, power may be transferred from power source1270to one or more of power sources370, or vice versa. Accordingly, power interfaces1272and1274may be positioned relative to compartments1202and1204such that power interfaces1272and1274align with a corresponding power interface372when earphones110and710are received in compartments1202and1204.

FIG.13Aillustrates an example configuration of earphone case130, consistent with the disclosed embodiments. As shown inFIG.13A, earphone case130may be configured to receive earphones110and710in compartments1202and1204, respectively. In this example, charging port640may be positioned relative to earphone710to enable a connection with power interface1274. For example, charging port640may be configured as two or more conductive plates configured to couple with corresponding pins or plates within compartment1204. Charging port640is shown by way of example inFIG.13Aand various other configurations may be used, including the configuration shown inFIG.6D. In some embodiments, earphone case130may include a first housing component1310and a second housing component1320. Housing components1310and1320may be movable relative to each other to allow earphones110and710to be inserted into and contained within earphone130. For example, earphone case130may include a hinge1330allowing relative rotational movement between housing components1310and1320. Earphone case130may further include a fastener (not shown inFIG.13A) to retain housing components1310and1320relative to each other and maintain earphone case130in a closed state. For example, this may include a mechanical latch, a magnetic latch, or various other forms of fasteners.

FIG.13Billustrates a view of the example configuration of earphone case130shown inFIG.13Afrom an alternate angle. As shown inFIG.13B, earphone case130may optionally include a camera and a microphone, which may correspond to image sensor1230and audio sensor1240, as shown. Image sensor1230and audio sensor1240may be positioned relative to earphone case130to facilitate the capture of images and/or audio signals by earphone case130. For example, when earphone case130is placed on a table, image sensor1230and audio sensor1240may be positioned to capture images and audio signals representative of an environment of earphone case130and/or user100, as described further below. In some embodiments, earphone case130may not include image and/or audio sensors.

FIG.14illustrates an example environment1400for processing audio signals using an earphone case, consistent with the disclosed embodiments. As described herein, earphone110may be configured to generate one or more processed audio signals based on captured audio signals, captured images, or both. For example, environment1400may include an individual1420, which may be the source of a sound1422. For example, sound1422may include a voice of individual1420. Environment1400may further include a sound1432, which may be a background or ambient noise from various additional sources. Microphone340may be configured to capture audio signals associated with sounds1422and1432and processor310may be configured to selectively condition one or both of sounds1422and1432, as described above. For example, selective conditioning may include amplifying audio signals determined to correspond to sound1422relative to sound1432. While two audio sources are shown in environment1400by way of example, sounds from any number of audio sources may be identified and selectively conditioned relative to each other, as described herein.

Consistent with the disclosed embodiments, some or all of processing of an audio signal captured using earphone110may be performed by processor1210. Accordingly, earphone110may be configured to capture an audio signal representative of sounds1422and1432and may transmit at least a portion of the captured audio signal to earphone case130. For example, this may include transmitting at least a portion of the captured audio signal using wireless transceiver360(i.e., a first communication component). Earphone case130(which may be located, for example, in one of positions1410,1412, or1414) may receive the transmitted portion of the captured audio signal using wireless transceiver1260(i.e., a second communication component), process the audio signal, and transmit at least a portion of the processed audio signal back to earphone110. Accordingly, earphone110may be configured to present an audio signal to an ear of user100, where at least a portion of the processing was performed using processor1210of earphone case130.

The processing of a captured audio signal may be distributed between earphone110, earphone710, and earphone case130in a wide variety of manners. For example, the processing may be divided evenly among processors, divided unevenly among processors (e.g., based on processing speed, processing techniques, available resources, etc.), performed redundantly by multiple processors, or the like. In some embodiments, processing tasks may be assigned and distributed by processor310. For example, processor310may analyze a captured audio signal, determine which processor(s) should perform one or more tasks, and may distribute portions of the audio signals to one or more of earphone710and/or earphone case130, if needed. Alternatively or additionally, processor1210may be configured to assign and distribute tasks. For example, processor310may transmit a captured audio signal to earphone case130and processor1210may distribute portions of the audio signals to one or more of earphones110and710, if needed. Accordingly, any configuration or sharing of processing capabilities may be used.

In some embodiments, processing allocations may be determined based, at least in part, on a type of analysis to be performed. For example, processor1210may have at least one performance characteristic that is advantageous relative to a corresponding performance characteristic of processor310. The performance characteristic may include a number of processing cores, a processing speed, a processor configuration (e.g., a graphics processor unit vs. a central processing unit), or the like. As another example, processor1210may have access to different algorithms, models, or other processing tools compared to processor310. For example, memory1220may store a trained machine learning model used to process audio signals, which may not be stored on memory320(e.g., due to size and/or performance constraints). Accordingly, processor1210may be better suited to handle certain tasks as compared to processor310and one or more tasks may be distributed to processor1210based on processor310being insufficient to perform the task. Alternatively, processors310and1210may be comparable in terms of processing capabilities, and tasks may be distributed to reduce the load on any one processor.

In some embodiments, processor1210may be configured to perform various specific tasks, as noted above. For example, processor1210may be configured to process lip movements of individuals, detect facial features, recognize an individual based on captured images (which may include recognizing facial features of individual1420, determine a relative direction of one or more sound sources, identify individuals based on stored voiceprints, or various other tasks described herein. While various distributions of tasks are provided above, it is to be understood that various other distributions of tasks may be performed. In some embodiments, the shared processing between earphone110and additional earphone710may not be split by discrete tasks and may be distributed in other ways. For example, as noted above, processor310may be a primary processor and may assign specific processing operations to processor1210and the result of the individual processing operations may be provided to processor310.

According to some embodiments, processor1210may only be used in certain situations. For example, processor310may be programmed to process captured audio signals and may initiate additional processing as needed. In some embodiments, additional processing may be initiated based on a quality or another aspect of a processed audio signal generated by processor310. For example, the quality or aspect may include a signal-to-noise ratio (SNR) of the processed audio signal, a volume level associated with at least one audio source represented in the captured audio signal (e.g., sound1422), or any other aspect of a processed audio signal that may indicate a processing quality.

In some embodiments, tasks may be allocated to processor1210based on the availability of other hardware of earphone case130, such as image sensor1230and audio sensor1240. For example, processor310may determine that additional information captured using image sensor1230and/or audio sensor1240may improve a quality of a processed audio signal and therefore may assign some or all of the processing to processor1210. In some embodiments, earphone110may not include an image sensor and thus may assign any processing based on analysis of captured images to processor1210. As another example, a field of view1402of image sensor330may be at least partially blocked or obstructed and processor1210may be used due to the unavailability or insufficiency of images captured using image sensor330. As another example, audio sensor1240may have at least one aspect that is improved relative to audio sensor330. For example, audio sensor1240may be a microphone array that is better suited to determined directionality of sounds, which may assist in separating distinct audio sources. Alternatively or additionally, audio sensor330may be at least partially blocked and therefore audio sensor1240may be needed. As another example, the use of multiple audio sensors340and1240may be used to distinguish sounds from different sources (e.g., by comparing variations in amplitudes in multiple signals, etc.). Accordingly, processor1210may be configured to process the captured audio signal based on an additional audio signal captured using audio sensor1240and/or at least one image captured using image sensor1230.

Some embodiments of the present disclosure may include advising or reminding user100to use earphone case130for processing audio signals. For example, in some situations, some or all of the processing capabilities of earphone case130may be at least temporarily unavailable or insufficient for the current conditions. In other words, earphone case130may not be in functional state for generating a processed audio signal. In some embodiments, processor1210may be unavailable entirely. For example, earphone case130may be out of range of earphone110and thus a communication signal between earphone case130and earphone110may be weak or may not be received. As another example, a battery level of power source1270may be low or depleted, which may prevent some or all processing capabilities of processor1210. As another example, a current state of earphone case130may similarly prevent some or all processing capabilities of processor1210. For example, this may occur when earphone case130is switched off, is in a power-saving mode, is damaged or in an error state, or the like.

In some embodiments, earphone case130may be deemed to not be in a functional state based on a specific functionality of earphone case130being unavailable. For example, despite being in an on state and within range of earphone110, various specific features of earphone case130may be disabled. As one example, image sensor1230and/or audio sensor may be at least partially blocked or obstructed, which may prevent sufficient image or audio data from being captured by earphone case130. For example, when earphone case is located in a bag1404of user100(position1410) or a pocket of user100(position1412), earphone case130may not be able to capture sufficient audio or image data. In some embodiments, this may be determined by processor1210. For example, processor1210may analyze one or more images captured by image sensor1230and/or one or more audio signals captured by audio sensor1240and determine that one or both of image sensor1230and/or audio sensor1240is not able to function properly (e.g., through dark or obstructed images, muffled audio signals, etc.). In some embodiments, this may be determined based on a specific audio source (e.g., individual1420) not being represented in captured audio or image data. For example, earphone case130may be placed in position1414but may be turned away from individual1420such that individual1420is not in a field of view1420of image sensor1230. Based on this assessment, processor1210may report to processor310that at least one function of earphone case130is unavailable or is not operational. In some embodiments, processor1210may assess the captured image and/or audio data based on a trigger event, such as a request from processor310to perform additional processing. Alternatively or additionally, an assessment of the audio and/or image data may be performed periodically to determine whether earphone case130is in a functional state for generating processed audio signals.

Based a determination that at least one function of earphone case130is unavailable (i.e., that earphone case is in a non-functional state for generating the processed audio signal), earphone110may cause a notification to use earphone case130to be presented to user100. The notification may be presented to user100in various ways. For example, this may include presenting vocal instructions, a chime, a tone, an alert, or various other audible notifications to an ear of user100via hearing interface350. In some embodiments, an audible notification may be presented from another device, such as computing device120. As another example, the notification may be a visual indicator. For example, a notification may be presented on a display of computing device120or another visual indicator component within system200. Accordingly, causing the notification to be presented may include transmitting a signal to another device of system200to cause the device to present the notification. As another example, the notification may be in the form of a tactile notification, such as a vibration, which may be presented by earphone110, computing device120, earphone case130(e.g., via feedback outputting units380,440, or1280), or any other device within system200.

In some embodiments, the notification may provide instructions to user100to take at least one action associated with earphone case130. In some embodiments, the notification may indicate a specific issue that was detected, such as a particular function being unavailable or impeded in some way, a signal strength or battery level being low, or any other assessed characteristics described above.FIG.15illustrates example notifications that may be presented to user100via computing device120, consistent with the disclosed embodiments. For example, a notification1510may be presented indicating that earphone case130is out of range. Referring toFIG.14, earphone case130may be in position1410or in any other location where a signal strength between earphone110and earphone case130may be weak or nonexistent. Accordingly, notification1510may include instructions for user100to move earphone case130closer to earphone110. As another example, a notification1520may be presented indicating that a battery level of earphone case130(e.g., a level of power source1270) is low or depleted. Accordingly, notification1520may include instructions to charge power source1270. As a further example, a notification1530may indicate that a subject is not within view of a camera of earphone case130. For example, processor1210may analyze an image captured using image sensor1230and may determine that individual1420is not represented (or represented clearly or fully) within the image. Accordingly, notification1530may further instruct user100to position earphone case such that the subject is within the field of view. Notifications1510,1520, and1530are provided by way of example, and various other forms of notifications may be presented, which may be based on a state of earphone case130. While notifications1510,1520, and1530are shown as visual notifications, it is to be understood that these notifications may equally be presented in other forms, such as an audible message presented in the ear of user100, a vibration, or the like. Additionally or alternatively, one or more notifications may be provided upon identification of a volume, clarity or another characteristic of a captured audio signal which indicates that additional capabilities or processing associated with earphone case130are required, and that it may be beneficial to make earphone case130available.

Based on the notification, user100may take at least one action associated with earphone case130. In some embodiments, this may include moving earphone case130to a location to facilitate generation of a processed audio signal. For example, based on one of notifications1510,1520, and/or1530, user100may move earphone case from one of positions1410or1412to position1414. Accordingly, earphone case130may be within range of earphone110and individual1420may be within the field of view1406of image sensor1230, as shown inFIG.14. As another example, user100may plug in earphone case (i.e., connect power interface1276to a power source), which may provide sufficient power to perform processing tasks by processor1210. Based on the action taken by user100, processor310may cause earphone case130to generate the processed audio signal, as described above.

According to some embodiments, the notification may be presented based on a determination that additional processing by processor1210is needed or may be advantageous. For example, processor310may identify at least one condition of environment140indicating that processing by earphone case130is required for generating a processed audio signal. In some embodiments, this may include analyzing image data and/or audio signals to identify the condition. As one example, the condition may be based on a noise level in the environment. For example, the noise level may be determined based on a captured audio signal. Based on a comparison of the noise level to a predefined threshold (e.g., a threshold noise level), processor310may determine that additional processing from processor1210would be helpful or needed. As another example, the condition may be based on a classification of the environment. In some embodiments, the classification may be a characterization of a type of an environment, such as “restaurant,” “meeting,” “home,” “vehicle,” or various other classifications, which may have different processing requirements. As another example, the classification may be based on a noise level, such as “loud,” “moderate,” “quiet,” etc. The classification may be based on other properties, such as a number of individuals in the environment, a number of speaking individuals, or the like.

The classification may be determined in various ways. In some embodiments, the classification may be based on analysis of images and/or audio signals captured by earphone110. For example, this may include application of an image analysis algorithm to detect and classify objects represented in one or more images. Based on the types of objects detected, earphone110may determine or estimate a type of environment. For example, based on a detection of chairs, tables, plates of food, or other objects in environment1400, processor310may classify environment1400as a restaurant. As another example, the classification may be based on application of a trained machine learning model. For example, a training set of image data may be input into a machine learning model, which may be labeled to include various classifications. Accordingly, a model may be trained to predict or determine classifications based on captured images. Alternatively or additionally, a model may be trained to predict or determine classifications based on captured audio. For example, the clanging of dishes and cutlery, background music, voices of many individuals, and other sounds may indicate a user is in a restaurant. Sounds such as a television, voices of a few individuals, a vacuum cleaner, of other sounds may indicate the user is at home. Similar to the model described above, a training set of audio data may be input into a machine learning model, which may be labeled to include various classifications. As a result, audio data captured using earphone110(and/or earphone case130may be input into the trained model, which may be trained to predict or determine classifications based on the captured audio. Consistent with the present disclosure, various types training or machine learning algorithms may be used, including a logistic regression, a linear regression, a regression, a random forest, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a cox proportional hazards regression model, a Naïve Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, or any other form of machine learning model or algorithm.

FIG.16is a flowchart showing an example process1600for generating processed audio signals, consistent with the disclosed embodiments. Process1600may be performed by at least one processing device of an earphone, such as processor310, and at least one processing device of an earphone case, such as processor1210. It is to be understood that throughout the present disclosure, the term “processor” is used as a shorthand for “at least one processor.” In other words, a processor may include one or more structures that perform logic operations whether such structures are collocated, connected, or dispersed. In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process1600. Further, process1600is not necessarily limited to the steps shown inFIG.16, and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process1600, including those described above with respect toFIGS.12,13A,13B,14, and15.

In step1610, process1600may include receiving a captured audio signal representative of sounds captured from an environment of at least one earphone. For example, the audio signal may include representations of sounds1422and1432from environment1400as described above. In some embodiments, the captured audio signal may be captured by at least one microphone of the at least one earphone. For example, step1610may include receiving an audio signal captured using audio sensor310. Accordingly, step1610may be performed using at least one first processor of the at least one earphone, such as processor310described above.

In step1620, process1600may include transmitting, using the at least one first processor, at least a portion of the captured audio signal via a first communication component of the at least one earphone. For example, step1610may include transmitting at least a portion of the captured audio signal via wireless transceiver360to earphone case130for additional processing, as described above. In some embodiments, the first communication component may be configured to communicate with various other devices of system200. For example, the at least one earphone may comprise a first earphone (e.g., earphone110) and a second earphone (e.g., earphone710) and the first communication component of the first earphone may be configured to communicate with the first communication component of the second earphone.

In some embodiments, the at least a portion of the captured audio signal may be transmitted based on a determination that additional processing is required, which may be based on an analysis of the captured audio signal. Accordingly, process1600may include generating an additional processed audio signal based on analysis of the at least a portion of captured audio signal using the first processor and determining, based on at least one aspect of the additional processed audio signal, whether to initiate additional processing. For example, the at least one aspect of the additional processed audio signal may include a signal-to-noise ratio of the additional processed audio signal or a volume level associated with at least one audio source represented in the captured audio signal, as described above. Step1620may include transmitting the at least a portion of the captured audio signal via the first communication component based on a determination to initiate additional processing.

In step1630, process1600may include receiving the at least a portion of the captured audio signal from the first communication component via a second communication component of the earphone case. For example, the at least a portion of the captured audio signal may be received using wireless transceiver1260of earphone case130, as described above. Accordingly, step1630may be performed using at least one second processor of earphone case, such as processor1210. The earphone case may include any of the various components described herein. For example, the earphone case may include at least one earphone compartment configured to receive the at least one earphone, such as compartments1202and1204described above. In some embodiments, the earphone case may further comprise a charging circuit for charging the at least one earphone when inserted into the at least one earphone compartment. For example, the charging circuit may include one or more of power interfaces1272and1274and power source1270.

In step1640, process1600may include generating, using the at least one second processor, a processed audio signal based on analysis of the at least a portion of captured audio signal. The processed audio signal may be generated in various ways, including the various forms of selective conditioning described herein. In some embodiments, generating the processed audio signal includes amplifying a voice of an individual represented in the captured audio signal relative to at least one additional sound. For example, this may include amplifying sound1422(which may include a voice of individual1420) relative to sound1432. In some embodiments, the voice of the individual may be amplified based on an input from the user. For example, the input may include a selection of a representation of the individual via a graphical user interface, such as graphical user interface1000described above.

According to some embodiments, the earphone case may include sensors for acquiring additional data used to generate the processed audio signal. For example, the earphone case may further comprise at least one additional microphone for capturing an additional audio signal. The additional microphone may correspond to audio sensor1240described above. The processed audio signal may further be generated based on analysis of at least a portion of the additional audio signal. In some embodiments, generating the processed audio signal may include determining a direction of at least one audio source represented in the captured audio signal based on the analysis of the at least a portion of the additional audio signal. For example, the at least one additional microphone comprises a microphone array, which may indicate a direction from which sounds are received relative to earphone case130.

As another example, the earphone case may further comprise at least one camera configured to capture a plurality of images from an environment of the earphone case. For example, the at least one camera may correspond to image sensor1230, as described above. The processed audio signal may further be generated based on analysis of at least one image of the plurality of images. For example, generating the processed audio signal may include amplifying a voice of an individual represented in the captured audio signal relative to at least one additional sound, as described above. In some embodiments, amplifying the voice of the individual may include identifying the individual based on the at least one image. For example, this may include recognizing individual1420, which may be based on facial features or other characteristics of individual1420.

In some embodiments, generating the processed audio signal may include application of a trained machine learning algorithm using the at least one second processor. For example, memory1220may include a machine learning model or algorithm trained to selectively condition the captured audio signal as described herein. For example, a training set of audio and/or image data may be input into a machine learning model, along with corresponding processed audio signals, which may be selectively conditioned to amplify a voice of one audio source relative to other sources. Accordingly, a model may be trained upon the training data to generate processed audio signals. In some embodiments the trained machine learning model may be trained to perform specific tasks, such as recognizing an individual based on facial features, voiceprints, etc., separating audio, enhancing audio, or various other tasks described herein. Consistent with the present disclosure, various training or machine learning algorithms may be used, including a logistic regression, a linear regression, a regression, a random forest, a K-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a cox proportional hazards regression model, a Naïve Bayes model, a Support Vector Machines (SVM) model, a gradient boosting algorithm, or any other form of machine learning model or algorithm.

In step1650, process1600may include transmitting, using the at least one second processor, at least a portion of the processed audio signal via the second communication component to the first communication component. For example, this may include transmitting the processed audio signal from wireless transceiver1260to wireless transceiver160, as described above. In some embodiments, process1600may further include presenting, using the at least one first processor, the at least a portion of the processed audio signal to an ear of a user of the at least one earphone. For example, this may include presenting the processed audio signal to an ear of user100using hearing interface component350.

In some embodiments, earphone110may be configured to determine whether additional processing from the earphone case is needed or desirable (i.e., is expected to improve a quality of processing the captured audio signal, etc.). Accordingly, process1600may further include identifying at least one condition of the environment of the at least one earphone. For example, the at least one condition may be based on a comparison of a noise level of the captured audio signal to a predetermined threshold, a classification of the environment of the at least one earphone, or various other forms of conditioning. Based on the at least one condition, process1600may include identifying that the earphone case is required for generating the processed audio signal; and causing a notification to use the earphone case to be presented to a user of the at least one earphone. The notification may be presented in various ways. For example, causing the notification to be presented may include transmitting a signal to at least one device associated with the at least one earphone; or providing at least one of a visual, auditory, or tactile signal to the user via the at least one earphone or by a paired device, as described above.

In some embodiments, the notification to use the earphone case may be caused to be presented based on whether the earphone case is in a functional state for generating the processed audio signal. For example, the notification may be presented to the user based on at least one of a signal strength of a signal received from the second communication component or a power level associated with the earphone case. Process1600may further include determining whether the earphone case has been restored to a functional state for generating the processed audio signal. For example, process1600may further include identifying that the user has placed the earphone case in location facilitating generation of the processed audio signal by the earphone case and causing the earphone case to generate the processed audio signal. In some embodiments, the earphone case may remind the user to use the earphone case if the functional state is not restored. For example, process1600may further include providing an additional notification to the user after a predetermined time has elapsed since providing the notification.

FIG.17is a flowchart showing another example process1700for generating processed audio signals, consistent with the disclosed embodiments. Process1700may be performed by at least one processing device of an earphone, such as processor310. In some embodiments, a non-transitory computer readable medium may contain instructions that when executed by a processor cause the processor to perform process1700. Further, process1700is not necessarily limited to the steps shown inFIG.17, and any steps or processes of the various embodiments described throughout the present disclosure may also be included in process1700, including those described above with respect toFIGS.12,13A,13B,14,15, and16.

In step1710, process1700may include receiving a captured audio signal representative of sounds captured from an environment of the at least one earphone by at least one microphone of the at least one earphone. For example, the audio signal may include representations of sounds1422and1432from environment1400as described above. In some embodiments, the captured audio signal may be captured by at least one microphone of the at least one earphone. For example, step1610may include receiving an audio signal captured using audio sensor310.

In step1720, process1700may include identifying at least one condition of the environment of the at least one earphone. Various types of conditions may be identified in step1720. As one example, the condition may be based on a noise level or a signal to noise ratio in the environment. As another example, the condition may be based on a classification of the environment, as described herein.

In step1730, process1700may include identifying that the earphone case is required for generating a processed audio signal based on the at least one condition. For example, this may include determining that the noise or the signal to noise level exceeds a threshold level, as described herein. As another example, this may include determining that the classification of the environment warrants additional processing by the earphone case.

In step1740, process1700may include determining whether the earphone case is in a functional state for generating the processed audio signal. A functional state may refer to a state in which some or all of the processing capabilities of earphone case are functional to generate a processed audio signal. In some embodiments, the functional state may be based on a state of earphone case130. For example, this may include a power level of earphone case130, a status or mode of earphone case130, a signal strength of earphone case130(e.g., indicating whether earphone case130is within range of earphone110), or the like. In some embodiments, the functional state may be based on specific functionality or features of earphone case130. For example, the functional state may be determined based on analysis of an image or audio data captured using earphone case130, as described above, which may indicate whether data captured image sensor1230or audio sensor1240would provide additional information that may be used to improve processing of the captured audio signal.

As indicated in step1750, based on a determination that the earphone case is not in the functional state, process1700may include causing a notification to use the earphone case to be presented to a user of the at least one earphone. The notification may be presented in various ways. For example, causing the notification to be presented may include transmitting a signal to at least one device associated with the at least one earphone; or providing at least one of a visual, auditory, or tactile signal to the user via the at least one earphone, as described above.

As indicated in step1760, based on a determination that the earphone is in the functional state, process1700may include transmitting at least a portion of the captured audio signal via the at least one first communication component to at least one second communication component of an earphone case. For example, step1760may include transmitting at least a portion of the captured audio signal via wireless transceiver360to earphone case130for additional processing. Accordingly, the earphone case may be configured to receive the at least a portion of the captured audio signal from the first communication component via a second communication component of the earphone case; generate a processed audio signal based on analysis of the at least a portion of captured audio signal; and transmit at least a portion of the processed audio signal via the second communication component to the first communication component, as described above.

The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Additionally, although aspects of the disclosed embodiments are described as being stored in memory, one skilled in the art will appreciate that these aspects can also be stored on other types of computer readable media, such as secondary storage devices, for example, hard disks or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray, Ultra HD Blu-ray, or other optical drive media.

Computer programs based on the written description and disclosed methods are within the skill of an experienced developer. The various programs or program modules can be created using any of the techniques known to one skilled in the art or can be designed in connection with existing software. For example, program sections or program modules can be designed in or by means of .Net Framework, .Net Compact Framework (and related languages, such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with included Java applets.

Moreover, while illustrative embodiments have been described herein, the scope of any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations and/or alterations as would be appreciated by those skilled in the art based on the present disclosure. The limitations in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application. The examples are to be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including by reordering steps and/or inserting or deleting steps. It is intended, therefore, that the specification and examples be considered as illustrative only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents.