Audio modification using interconnected electronic devices

Systems and methods are provided for reducing unwanted noise in an electronic audio signal, wherein a computing device having a microphone is configured to receive signals from a sensor on an external device such as a camera, second microphone, or movement sensor. The signals from the sensor are used to identify sound information or characteristics of sounds made by a source of noise, and the audio signal of the microphone is modified to reduce unwanted sounds based on that sound information or based on sounds identified a second audio signal obtained by the second microphone, thereby improving teleconference and video conference audio quality and removing distracting noises from transmitted audio output.

FIELD

The described embodiments relate generally to audio modification to remove unwanted sounds. More particularly, the present embodiments relate to using multiple interconnected electronic devices to improve unwanted noise reduction.

BACKGROUND

Teleconferences and video conferences are becoming ever more popular mechanisms for communicating. Many portable computer devices, such as laptops, tablet computers, and smartphones today have built-in microphones usable for these purposes. In addition, many portable computer devices have built-in cameras (or can easily have an inexpensive external camera, such as a web cam, added). This allows for very low cost, highly prevalent participation in teleconferences and video conferences.

It is common for background noises to occur during the conference, such as participants typing on the device being used for the conference. For example, a participant may be taking notes about the conference or multi-tasking while talking or while listening to others talk. With the physical proximity of the keyboard on the portable computer device to a microphone that may also be on the portable computer device, the microphone can easily pick up noise from the keystrokes and transmit the noise to the conference, causing distraction and annoyance to other participants.

Although many products and schemes have been devised for noise canceling, including specifically canceling noises produced by keyboard typing in computer teleconferencing, these systems leave often lack precision and accuracy when canceling the noise. Furthermore, audio recordings of many other kinds, such as recordings of musical instruments, can be improved by removing unwanted sounds. There is, therefore, a constant need for improvements to audio modification systems and techniques.

SUMMARY

One aspect of the present disclosure relates to a computing device for managing teleconferencing. The computing device can include a processor and a memory device configured for electrical communication with the processor. The memory device can include instructions encoded thereon that, when executed by the processor, cause the processor to receive an audio signal from a microphone of a source computer, receive a sensor signal from at least one of a camera, a movement sensor, a position sensor, or a second microphone at the source computer, detect, using the sensor signal, a source of a sound in the audio signal of the microphone, modify the audio signal to reduce the sound in the audio signal, and send the modified audio signal to a destination computer.

In some examples, detecting the source can include identifying a computer input device in an image obtained from the camera, and the sound can include a noise produced by a person using the computer input device. The instructions can further cause the processor to detect a position of a user relative to the source computer, wherein the audio signal can be modified based on the position of the user relative to the source computer. Detecting the source can include detecting a movement or change in position of the source computer via the movement sensor or the position sensor. The camera, the movement sensor, the position sensor, or the second microphone can be attached to the source computer. In some examples, the camera, the movement sensor, the position sensor, or the second microphone can be part of a device separate from, and in electrical communication with, the source computer.

Another aspect of the disclosure relates to a method of managing sounds and noise while teleconferencing. The method can include recording an audio signal via a microphone of a source computer, sensing a sound source via a sensor including a camera, a movement sensor, or a second microphone, detecting a wanted sound in the audio signal and an unwanted sound in the audio signal, wherein the wanted sound is created by the sound source detected via the sensor, amplifying the wanted sound in the audio signal relative to the unwanted sound, and transmitting the amplified audio signal to a destination computer.

In some embodiments, detecting the sound source includes detecting a person via the sensor, and wherein the wanted sound includes a vocal sound and the unwanted sound includes a non-vocal sound. The camera, the movement sensor, or the second microphone can be part of a device separate from, and in electrical communication with, the source computer. Detecting the sound source can include identifying a computer input device in an image obtained from the camera, and the unwanted sound can include a noise produced by a person using the computer input device.

In some embodiments, the method can further include detecting a position of a user relative to the source computer via the sensor, wherein the wanted sound is amplified based on the position of the user relative to the source computer. In some embodiments, detecting the sound source includes detecting a movement or change in position of the source computer via the movement sensor.

Another aspect of the disclosure relates to a computing device including an imaging device, a microphone, a processor in electronic communication with the imaging device and with the microphone, and a memory device in electronic communication with the processor. The memory device can include instructions encoded thereon that, when executed by the processor, cause the computing device to obtain an image via the imaging device, identify a source of a target noise in the image, receive an audio signal produced by the microphone, and modify the audio signal to change a representation of the target noise in the audio signal.

Modifying the audio signal can include at least partially canceling the representation of the target noise in the audio signal. Modifying the audio signal can also include isolating the representation of the target noise in the audio signal. In some examples, isolating the representation of the target noise includes beamforming microphones to the source of the target noise. Identifying the source can include identifying an object in the image. The object can include a body part of a person. The target noise can include a human vocal sound, and identifying the source can include detecting a vocalizing action by a person in the image.

Yet another aspect of the disclosure relates to a system for reducing unwanted noise in an electronic audio signal, with the system including a computing device including a processor, a memory device, and a microphone, and an electronic device in electrical communication with and separate from the computing device, the electronic device including a sensor. The memory device can include electronic instructions encoded thereon that, when executed by the processor, cause the computing device to: detect a source of a target noise using the sensor of the electronic device, receive an audio signal produced by the microphone of the computing device, with the audio signal including a representation of the target noise, and modify the audio signal to reduce the representation of the target noise in the audio signal.

In some examples, the computing device includes a keyboard, the target noise is a sound originating from the keyboard, the representation of the target noise is a recording of the target noise, and modifying the audio signal includes at least partially canceling out the recording of the target noise in the audio signal. The sensor can include an imaging device, and detecting the source of the target noise can include detecting an object in an image sensed by the imaging device. The sensor can include a second microphone configured to detect the target noise, and detecting production of the target noise can include receiving an audio signal produced by the second microphone including a second representation of the target noise. The sensor can be configured to detect a position or a movement of the electronic device, and detecting production of the target noise can include detecting a change in position of the electronic device or a movement of the electronic device via the sensor. The electronic device can include a wearable electronic device. The electronic device can include a peripheral input device for the computing device.

DETAILED DESCRIPTION

The following disclosure relates to using microphones, cameras, position and movement sensors, and related devices to identify unwanted sounds in an audio signal, or to identify sources of unwanted sounds in an audio signal, image, or position/motion signal, and to modify the audio signal or mute recording devices to reduce the occurrence, volume, or prevalence of unwanted sound in an output audio signal. Thus, by using principles of the present disclosure, unwanted sounds can be removed from audio signals recorded for teleconferencing, video conferencing, musical recordings, voice messages, and related activities.

Although conventional systems and methods have been devised that include actively canceling bands of frequencies in an audio signal, such as in active noise-canceling headphones which at least partially inverse a recorded audio signal and provide the modified signal to the user via a speaker, these systems and methods do not perform well in eliminating unique sounds and noises that fall outside predefined frequency limits. Additionally, although some systems and methods have been proposed that claim to cancel noise related to specific waveforms, such as keyboard typing sounds, detecting the production of the sound is generally reactive or based on getting a direct signal from the source of the sound, such as by detecting that a keyboard is being operated due to switches of the keyboard itself being triggered.

Conventional systems and methods can be improved through principles and aspects of the present disclosure which relate to using a system of devices that coordinate using multiple different sensors and/or multiple different types of sensors on one or more device to better identify, isolate, and reduce sounds in an audio signal. Additionally, aspects of the present disclosure relate to anticipating the appearance of sounds in an audio signal to preemptively remove unwanted noises or to provide information such as warnings to users of the systems described herein.

Some embodiments can include a computing device for managing teleconferencing, such as a server or client device that is configured to receive an audio signal from a microphone of the source computer and to receive a sensor signal from a separate sensor such as a camera, a movement sensor, a position sensor, or second microphone that is either part of, or in the vicinity of, the source computer. The sensor signal can come from electronic devices that are commonly used in the environment of a teleconferencing participant, such as a smart phone, a tablet computer, a smart watch or other wearable smart device, a headset or headphone device, a smart speaker or other recording device, related devices, and combinations thereof. Thus, cameras and other sensors on these nearby devices can be used to help collect signals, images, and other information in the environment of the participant to identify and remove unwanted sounds more effectively and optimally than could be done with a single device. The modified audio signal can then be sent to other devices, such as a destination computer, and the participants at the destination computer can enjoy clearer, less-distracting communication with those at the source computer.

A camera or other image sensor can be used to reduce unwanted noises, object, person, and shape recognition techniques by analyzing images from the camera to determine sources of unwanted sounds by their appearance, by their movement in images or videos, by their distance from the camera, etc. For example, in one embodiment, the camera can be used to observe and determine whether a participant's mouth is moving or not, and an audio signal recorded by the participant's device can be modified (e.g., audio can be muted when the mouth is not moving and unmuted when the mouth is moving). Furthermore, the camera can observe the position and/or orientation of the participant to enable the system to intelligently determine whether the participant intends to provide input to the microphone (e.g., is facing the microphone) or not so that an intentional communication can be reduced or muted entirely.

In another example, the camera can be used to observe the position and condition of an object, such as a computer input device (e.g., a peripheral input device), to determine whether a user is typing, clicking a mouse, adjusting a microphone, etc., and the audio signal can be modified by muting or unmuting a microphone to avoid the sound or by filtering out/canceling out certain waveforms or frequencies corresponding to noise produced by the object present in the camera image. In this case, the system can access a database containing representative recordings of sounds made by the object and can thereby effectively identify and cancel out those sounds when they are recorded by the primary microphone, thereby enabling noise cancellation of specific sounds using a camera to identify which sounds need to be canceled.

In embodiments using multiple microphones, a primary microphone signal can be recorded using a computing device, and a secondary microphone signal can be recorded using a separate device in the environment of the computing device. The separate device, such as a smart phone or wearable device in the same room as the computing device, can obtain the secondary microphone signal with waveforms that are present in the primary microphone signal, but at different amplitudes and, potentially, different frequencies. The differences between the multiple microphone signals can be analyzed by the computing device to identify and remove specific unwanted sounds (or to beamform microphones to isolate wanted sounds coming from a target source (e.g., the user's face)). Isolating wanted sounds coming from a sound source can comprise amplifying those sounds relative to other, unwanted sounds recorded in the environment of the sound source, such as by attenuating frequencies other than those in the wanted sounds, increasing the volume or amplitude of the waveforms or frequencies corresponding to the wanted sounds, similar methods, and combinations thereof.

In embodiments using position or movement sensors, movements or changes in the position of a computing device or a secondary device (e.g., a wearable device) can be used to determine when certain unwanted noises are being made in the user's environment. For example, accelerometers in a smart watch can output signals suggesting that a user is typing on a keyboard or raising his or her elbow to sneeze, and that data can be used to predict and reduce the volume or prevalence of unwanted noises that correspond to that activity detected.

FIG.1is an illustration of a video conferencing environment100showing aspects of the present disclosure. It will be understood that although a video conferencing environment100is shown inFIG.1, principles and aspects of the present disclosure can be applied to many different settings in which audio recordings are being made and/or transmitted, such as in teleconferencing (including telephone calls), studio recording (e.g., musical recordings), live recording, filmmaking, telepresence interacting, robot control, related settings and areas of application, and combinations thereof. The same is true of other embodiments disclosed in connection with the other figures. Furthermore, as used herein, a device (e.g., a sensor) comprising at least one of a first option (e.g., a camera), a second option (e.g., a movement sensor), or a third option (e.g., a second microphone) should be understood as referring to a device that can include one of each listed option (e.g., only one of the first option, only one of the second option, or only one of the third option), multiple of a single listed option (e.g., two or more of the first option), two options simultaneously (e.g., one of the first option and one of the second option), or combination thereof (e.g., two of the first option and one of the second option).

As shown in the video conferencing environment100, a user102(i.e., a conference participant) interacts with a computing device104. The computing device104can include a set of computer input devices (e.g., keyboard116and touchpad118), a display120, and a camera122(or other imaging device).

As the user102interacts with the computing device104, the user102can vocalize or make other vocalizing actions or noise-making sounds with his or her body, symbolically represented as sound124, and can make sounds while interacting with items in the environment100, such as the keyboard116and touchpad118, symbolically represented as sound126. A microphone used by the user102, such as a microphone of the computing device104(seeFIG.2), can record the sounds124,126and can produce an audio signal, schematically represented by waveform128. The signal can send the recorded waveform128to another device, such as another computing device (i.e., a destination computing device) where another user can listen to the sound represented by the waveform128at another location via a loudspeaker (e.g., a loudspeaker on the other computing device). Thus, the user102can send an audible message to one or more other users via the computing device104, such as in a teleconference. Other sounds, such as non-vocal sounds (e.g., noises made by other body parts of the speaker, other devices in the vicinity of the speaker, etc.) can also be recorded in the teleconference. These other sounds, such as non-vocal sounds that are unwanted in the teleconference, can be detected and removed, or made less prominent relative to the vocal sounds in the recording.

Additionally, the camera122of the computing device104can obtain an image130or a series of images (e.g., a still image or video recording) of the user102, other people, animals, devices (e.g.,106,108,110,112,116,118), and other objects (e.g., inanimate objects or animate objects that are not in electrical communication with the computing device104) in the environment100. While videoconferencing, the camera122can therefore obtain an image130or video feed that is transferred to other users.

FIG.2illustrates a schematic representation of a system200for reducing unwanted noise in an electronic audio signal. The system200can include a computing device202, such as, for example, the computing device104ofFIG.1or the computing system1800ofFIG.18. The computing device202can be referred to as a source device or source computer, and devices to which audio signals are sent by the computing device202can be referred to as destination devices or destination computers. The computing device202can include computing components such as those described in connection withFIG.18below (e.g., a processor201or1802). Thus, only a limited number of components of the computing device202are shown in the block diagram ofFIG.2. The computing device202can include a database204(e.g., on a memory device), a network connection206, a microphone208, a camera210(e.g., camera122), and a keyboard212(e.g., keyboard116).

The microphone208can transduce sound waves from the environment in which a user214is located near the computing device202, as indicated by arrow216. Thus, the user can vocalize or otherwise make noise near the computing device202to record a waveform using the microphone208. The recorded waveform can be converted and sent to other computing devices via the network connection206, as indicated by arrow218.

Sometimes, sounds made by the user214or sounds in the environment of the user214are unwanted or undesirable to send to other computing devices. For example, the user214can operate the keyboard212while the microphone208is actively recording the sound in the environment, as indicated by arrow220, and the noise made by the keyboard212can be distracting or otherwise obtrusive to listeners at the other computing devices. In these situations, the computing device202can be configured to identify the unwanted noises (e.g., the sounds recorded as represented by arrow220) in the recorded waveform using the camera210.

The camera210can be positioned and oriented on or near the computing device202in a manner configured to observe and record images (as represented by arrow222) of typical sound-producing objects, people, and animals in the surroundings of the computing device202. The camera210can therefore, in some embodiments, face toward the user's face or hands, toward the keyboard212, toward another computer input device or external device, or toward another typical source of unwanted sounds. As represented by arrow224, the camera210can receive images of the keyboard212in this example system200.

FIG.3shows an example image300captured by a camera of the present disclosure (e.g., camera122or210). The image300can have a border, frame, or other outer limit within which the camera is capable of discerning light that enters a sensor in the camera, as represented by the generally rectangular shape of image300. In other embodiments, the image300can include alternate aspect ratios and shapes (e.g., rounded rectangular, square, wide angle/fisheye, elliptical, or circular).

The image300can include representations of people and objects within the view of the camera, such as an image of the user302or an image of a body part or appendage of the user (e.g., an image of her mouth304or an image of her hand306). The image300can also include representations of other people (e.g., conversing couple308) and objects (e.g., fan310) within the field of view of the camera. Accordingly, information from the camera in the image300can be provided to a processor of the computing device202for analysis.

FIG.4is a flow diagram showing a method400of processing and analyzing images and audio signals in order to reduce unwanted noise in an audio recording. In this method400, a computing device can receive an image and audio signal, as indicated in block402. For example, the computing device can receive an image (e.g.,300) from a camera (e.g.,122and210) and can receive an audio signal from a microphone (e.g.,208). The camera or microphone can be part of the computing device operating method400or can be transmitted to the computing device from a separate computing device or other electronic device having the camera or microphone.

As indicated in block404, the computing device can identify source of a noise in the image. For instance, as shown inFIG.3, the image300can include shapes, colors, tones, etc. that are recorded representations of people and things in the field of view of the camera. Thus, while performing block404, the computing device can analyze the information in the image300and correlate the information in the image with people and things that are sources of noises. For example, the computing device can use object recognition techniques known in the art (e.g., edge detection, shape detection, etc.) to identify what certain shapes and other image information in the image300represent. A face recognition algorithm can be used to identify a user214from the image of the user302or other people from the representation of the conversing couple308, and a shape recognition algorithm can be used to identify a mouth or hand from the image representations of the mouth and hand304,306or to identify a fan from the image representation of a fan310. Additionally, performance of block404can include analyzing a video or series of images to determine that people or things in the video or series of images are generating sound based on their movements, such as by determining via the computing device that a mouth of a person is moving in a manner correlated to a vocalizing movement, that the fan is on and spinning, that the hand of the user is touching a keyboard in a typing manner, etc.

The method400can further include identifying sound information (e.g., a waveform or sound pattern) correlated with the source of the noise identified, as indicated in block406. Identifying the sound information can include accessing a database (e.g.,204or a network-connected database) that stores recorded sounds or other sound information that is representative of various objects. Thus, identifying the sound information can include identifying one or more recorded sounds in the database that correspond to the object or person identified in connection with block404. For example, such recorded sound information is shown and discussed in connection withFIG.6below. The waveform or sound pattern identified in block406can be a recording of the source of block404or can be a set of sound properties (e.g., frequency, rhythm, harmonization, modulation, etc.) that generally define noises made by objects similar to the source of block404. Thus, the waveform or sound pattern does not necessarily have to be an exact representation or recording of the source of block404and can be an approximation or similar representation thereof. Additionally, if multiple different sources of noises are detected in the same image, the computing device can determine sound information for each of the sources, such as the different waveforms312shown inFIG.3that correspond to different noise sources (e.g.,304,306,308,310) in image300.

The method400can further include modifying the audio signal of block402using the sound information identified in block406. For instance, the computing device can analyze the audio signal of block402to identify waveforms and sound patterns that indicate the recorded presence of a noise produced by the source identified in block404(i.e., a target noise). In some examples, the audio signal can include sound information (e.g., a pattern or frequency) that is similar to or a copy of the target noise within a recorded time span in the audio signal, and the computing device can modify the audio signal within that recorded time span to change the representation of the target noise, as indicated in block408.

Modifying the audio signal can include reducing the volume or amplitude of a waveform or set of frequencies in the audio signal to make the target noise less prevalent or noticeable to a listener of the audio signal at a destination computer or at the computing device when the audio signal is played back. For example, as shown inFIG.5, the audio signal (e.g., waveform128) can include various frequencies and amplitudes recorded over time.FIG.6shows a representation of a waveform or sound pattern that correlates to a noise made by a particular object (as identified and determined in block406). Thus, in block408, the computing device can analyze the audio signal ofFIG.5to detect the presence of a waveform similar to or matching the waveform ofFIG.6. In this example, the waveform ofFIG.6is identified within time span500. Accordingly, the computing device can modify the audio signal within time span500to attenuate certain frequencies (or all frequencies) to minimize or eliminate the appearance of the target noise in the modified audio signal, as shown inFIG.7. Similarly, a microphone can be muted when those sounds occur or those sounds can be removed when the recorded audio signal is sent to another device. In this way, the computing device can produce a modified audio signal that is less distracting and contains fewer or quieter unwanted noises, thereby leading to an improved user experience for the presenter in a teleconference or videoconference and for the viewers or listeners as well.

Referring again toFIG.4, in a related embodiment, the computing device can modify the audio signal in block408by applying a filter to the entire audio signal based on the sound information of block406. Thus, rather than identifying a particular waveform in a particular span of time of the recorded audio signal, the sound information of block406can include some properties to generally modify the entire audio signal. For example, if the image contains a noise-making object such as a fan, the frequencies associated with a fan can be attenuated and removed from the entire audio signal, even when the fan is not detected or at times before the fan has been detected by the camera, as opposed to only removing those frequencies when the fan is visible to the camera or as opposed to only removing those frequencies when a particular fan-representative waveform is identified in the audio signal. Comparably, if a keyboard is identified in the camera image, any typing sound patterns or correlated frequencies in the audio signal can be detected and removed, whether or not the keyboard is visible to the camera at the time the typing sounds occur.

FIG.8shows another environment800similar to environment100in which systems and devices of the present disclosure are shown. In this environment800, reference numbers are repeated for elements already described in connection with environment100. In addition to those elements, the environment800can include an external device802having an image sensor (e.g., a security camera, webcam, smart phone (e.g.,1110inFIG.11), second computing device, tablet computer, similar devices, and combinations thereof) configured to collect image data from the environment800separate from the computing device104, such as image804. The external device802can therefore be positioned and oriented to obtain a different field of view or different image information as compared to the camera122of the computing device104. Furthermore, in some examples the computing device104lacks a camera122, and the external device802is the only image-capturing device in the environment800. The external device802can be in electronic communication with the computing device104to provide its image information804to the computing device. In some examples, the external device can be in electronic communication with the network to which the computing device104is in electronic communication so that the image804can be relayed to the computing device104via the network. For example, the devices can communicate via network interfaces1812and a network1805, as described in connection withFIG.18below.

FIG.9shows a system900corresponding to environment800and similar to system200. In this system900, reference numbers are repeated for elements already described in connection with system200. An external device902(e.g., external device802) has a camera904or other image-capturing device. The camera904can be configured to receive image information from the environment (e.g.,800), such as by having the keyboard212or the user214within the field of view of the camera904, as indicated by arrows906and908, respectively. As discussed in connection withFIG.2, microphone208can receive audio signals, as suggested by arrow216. The external device902can output a signal, as indicated by arrow910, that is transferred to the computing device202. In some examples, the signal from the external device902is transferred to the computing device202via the network connection206. Thus, the devices202and902can be in electrical communication with each other.

FIG.10shows an example image1000collected from a camera of an external device, such as, for example, camera904of external device902. Similar to image300, the image1000can include shapes, lines, and other image information representing people and objects within the field of view of the camera. Additionally, the image1000and can view the user and her computing device from a different angle than from the perspective of a camera on the computing device itself (e.g.,122), which can potentially allow the image1000to detect image information of parts of the computing device that would otherwise not be viewable by a camera on the computing device. For example, cameras on computing devices are often positioned adjacent to, and in plane with, a display screen, thereby making the display screen impossible to view with the camera. Using an external device, the image1000can include image information showing the display screen, applications being operated on the display screen, another camera on the computing device, a backside of the camera or display housing, etc. Accordingly, image information in the image1000of the external device can be used in place of, or in addition to, image information obtained by a camera of the computing device. This image information can allow the computing device to identify other sources of noises external to the computing device, and identify sound information associated with those sources such as the examples of sound information1002shown inFIG.10. This image information can be used in connection with method400to modify audio signals using the sound information detected based on sources of noise in the image1000, as described in connection withFIG.4.

Additionally, using image information from an external device can facilitate determining distances between the user and the computing device or between the user and other noise-making objects in the environment of the user. Thus, in some embodiments, performance of block408can include modifying the audio signal based on how far apart the user or other noisemaking objects are from the microphone or from each other. For example, the audio signal can be less attenuated for certain frequencies if it can be determined that a source of an noise that makes those frequencies is far away from the microphone obtaining the audio signal, thereby limiting the amount of attenuation that would unnecessarily interfere with the other sounds recorded in the audio signal. For noise sources that are closer to the microphone, sounds can be reduced, muted, or canceled more aggressively to help ensure that a user's voice content in the audio signal is preserved.

FIG.11shows yet another environment1100similar to environments100and800in which systems and devices of the present disclosure are shown. In this environment1100, reference numbers are repeated for elements already described in connection with environment100. In addition to those elements, the environment1100can include external devices such as, for example, a set of wearable devices being worn by the user102, including, in this example, a smart wristwatch1106or headphones1108. Other devices in the environment1100can include an external or secondary computing device such as a smart phone or tablet computing device1110, a “smart speaker”1112, another computing device, another recording device, related devices, and combinations thereof.

The other devices, such as wristwatch1106, headphones1108, tablet device1110, and smart speaker1112, can have their own microphones that are separate from the microphone(s) of the computing device104. Thus, as shown with smart speaker1112, for example, the sounds124,126can be recorded by the smart speaker1112and converted into a waveform1132. In some embodiments, the waveform1132can be sent to the computing device104. Similarly, audio data similar to waveform1132can be collected by other devices (e.g.,1106,1108,1110) in the environment1100.

The other devices can have image-capturing capability in the environment1100, such as a tablet device1110, headset, or visor worn by the user102. Those devices can capture an image (or series of images/video) that can be used similar to the image804, as described above. Furthermore, an imaging device such as camera802can be used in conjunction with the other devices shown inFIG.11.

FIG.12shows a system1200corresponding to environment1100and similar to systems200and900. In this embodiment, reference numbers are repeated for elements already described in connection with systems200and900. In some embodiments, elements from systems200and900can be incorporated into system1200. For example, the computing device202can include a camera (e.g.,210) configured to perform functions described above in connection withFIG.2, and the external device1202can include a camera (e.g.,904) configured to perform functions described above in connection withFIG.9. The external device1202can output a signal, as indicated by arrow1216, that is transferred to the computing device202. In some examples, the signal from the external device1202is transferred to the computing device202via the network connection206. Also, in some examples, the computing device202can be used to control the external device1202(or external device902).

An external device1202(e.g., one of external devices1110,1112or wearable devices1106,1108) can include a microphone1208that is separate from the microphone208of the computing device202. Thus, the microphone208can be referred to as a first microphone, and microphone1208can be referred to as a second microphone, an external microphone, or an environmental microphone.

The second microphone1208can be used to record audio and to create and produce an additional or secondary audio signal that is different from the main or primary audio signal generated by the microphone208. Thus, as schematically shown inFIG.13, the primary or first microphone (e.g.,208) can produce a first audio signal.FIG.12shows an example illustration of this action, as indicated by arrows216and220, wherein sounds from the keyboard212and user214are recorded by the first microphone208. Simultaneously, the additional or second microphone1208of the external device1202can be positioned within the environment of the keyboard212and user214and can produce a second audio signal, as schematically shown inFIG.14and as indicated by arrows1210and1212inFIG.12. The first audio signal ofFIG.13and the second audio signal ofFIG.14can be used to produce a modified audio signal, as shown inFIG.15, as described below.

FIG.16is a flow diagram showing a method1600of processing and analyzing multiple audio signals in order to reduce unwanted noise in an audio recording. In this method1600, a computing device can receive at least two audio signals, as indicated in block1602. For example, the computing device can receive a first audio signal (e.g., as shown inFIG.13and via arrow220) from a first microphone (e.g.,208) and can receive a second audio signal from a second microphone (e.g.,1208). The first microphone can be part of the computing device operating the method1600, and the second microphone can be part of an external device (e.g.,1202) configured to electronically communicate with the computing device operating the method1600.

As indicated in block1604, the computing device can identify a source of a noise using the second audio signal. For instance, as shown inFIG.14, a waveform can include recorded representations of sounds made by people and things (e.g., a keyboard212) in the sensing range of the second microphone. Thus, while performing block1604, the computing device can analyze the information in the second audio signal and correlate the audio information in the second audio signal with people and things that are sources of noises. For example, the computing device can use sound recognition techniques known in the art (e.g., music recognition, speech recognition, acoustic fingerprinting, spectrogram data processing, feature extraction, classification algorithms, etc.) to identify what sources of noise generate certain forms, frequencies, rhythms, and other audio information represented in the second audio signal. In some examples, a voice recognition algorithm can be used to identify a user214from the vocal sounds of a person's speech recorded by the second microphone, and an acoustic fingerprint recognition algorithm can be used to identify a typing sound on a keyboard in the second audio signal.

In some embodiments, an audio signal can be provided to the computing device for the computing device to record/“learn” and compare to sound patterns in the first and second audio signals. Additionally, in some embodiments, performance of block1604can include analyzing the second audio signal to recognize non-vocal sounds made by a particular user or produced around the user, such as by detecting a particular user's typing cadence, a coughing sound, common sounds in their environment (e.g., a sound of their dog barking), etc. Accordingly, performing block1604can include tracking the occurrence of sounds in the first or second audio signals over time to help identify sources of noise as they occur for specific users over time.

The computing device can analyze the waveforms recorded by the first and second microphones and detect a representation of a target noise (e.g., sound pattern1400). The target noise can occur in both audio signals of the first and second microphones, wherein the sound pattern1400occurs during a span of time that overlaps the overall span of time recorded by the first microphone (i.e., where sound pattern1300occurs inFIG.13). The representation of the target noise obtained by the second microphone can beneficially be louder or more prominent when recorded by the second microphone (i.e., in pattern1400) as compared to the target noise recorded by the first microphone (i.e., in pattern1300) or relative to other sounds recorded by the second microphone (i.e., the rest of the audio signal outside pattern1400inFIG.14). In this case, the computing device can have a clearer signal with which to identify the representation of the target noise in the overall audio recordings of the first and second microphones. The clearer signal can allow the computing device to more accurately identify the source of the target noise. Accordingly, in some embodiments, the method1600can include positioning the second microphone in the environment of the user in a position more likely to record an unwanted noise in the environment relative to the first microphone, such as by positioning the second microphone closer to a keyboard where typing is expected to take place or closer to a window where nearby traffic is anticipated to make distracting sounds. The first microphone can, in that case, be positioned relatively closer to a primary audio source such as by being closer to the intended speaker in a teleconference to accentuate wanted sounds relative to unwanted sounds in the first microphone's audio signal.

The method1600can further include identifying sound information (e.g., a waveform, frequency, rhythm, or sound pattern) correlated with the source of the noise identified, as indicated in block1606, which is shown in broken lines to indicate that it is an optional step to be performed in some embodiments. Identifying the sound information can include accessing a database (e.g.,204or a network-connected database of information that is accessible using the network connection206) that stores recorded sounds or other sound information that is representative of the noise source. Thus, identifying the sound information can include identifying one or more recorded sounds in the database that correspond to the noise source identified in connection with block1604. For example, such recorded sound information is shown and discussed in connection withFIG.6. The waveform or sound pattern identified in block1606can be a recording of the source of block1604or can be a set of sound properties (e.g., frequency, rhythm, harmonization, modulation, etc.) that generally define noises made by objects similar to the source of block1604. Thus, the waveform or sound pattern does not necessarily have to be an exact representation or recording of the source of block1604and can be an approximation or similar representation thereof. Additionally, if multiple different sources of noises are detected in the same sound recording, the computing device can determine sound information for each of the sources.

The method1600can further include modifying the audio signal of block1602based on the source identified in block1604or the sound information identified in block1606, as indicated in block1608. For instance, the computing device can correlate sounds made by the source of noise identified in block1604based on their appearance in the recording made by the second microphone (after identifying pattern1400) and then reducing or attenuating those sounds in the recording made by the first microphone (i.e., within the time period of pattern1300), as indicated by modified sound pattern1500inFIG.15.

Furthermore, in some embodiments, the audio signal can include sound information (e.g., a pattern or frequency) that has characteristics similar to, or a copy of, the sound information determined in block1606within a recorded time span in the first or second audio signal, and the computing device can modify the first audio signal within that recorded time span to change the representation of the target noise, as indicated in pattern1500.

In any embodiment, modifying the audio signal can include reducing the volume or amplitude of a waveform or set of frequencies in the audio signal to make the target noise (or other noises similar thereto) less prevalent or noticeable to a listener of the audio signal at a destination computer or at the computing device when the audio signal is played back. For example, as shown inFIG.13, the audio signal (e.g., waveform128inFIG.11) can include various frequencies and amplitudes recorded over time.FIG.14shows a representation of a waveform or sound pattern recorded by the second microphone (e.g., waveform1132). Thus, in block1608, the computing device can analyze the audio signal ofFIG.13to detect the presence of a waveform similar to or matching a representation of a target noise (i.e.,1400) that appears in the waveform ofFIG.14. Accordingly, the computing device can modify the audio signal1300within the time span correlating to pattern1400to attenuate certain frequencies (or all frequencies) to minimize or eliminate the appearance of the target noise in the modified audio signal, as shown by pattern1500inFIG.15. In other embodiments, the first and second microphones can be used for beamforming to isolate and enhance or increase the volume of wanted sounds relative to unwanted sounds. For example, vocalizations common to both of the audio signals of the microphones can be isolated by canceling or muting sounds that are not determined to be part of the vocalizations. The system can thereby focus on transmitting the vocalizations that are typically the most important part of the audio signal for a teleconference while allowing the other unwanted sounds to fade and recede in the teleconference. Furthermore, as a result of any of these operations, the computing device can produce a modified audio signal that is less distracting and contains fewer or quieter unwanted noises, thereby leading to an improved user experience for the presenter in a teleconference or videoconference and for the viewers or listeners as well.

Referring again toFIG.16, in a related embodiment, the computing device can modify the audio signal in block1608by applying a filter to the entire audio signal based on the sound information of block1606. Thus, rather than identifying a particular waveform in a particular span of time of the recorded audio signal, the sound information of block1606can include some properties to generally modify throughout the audio signal. For example, if the second audio signal contains recorded noises correlated with a noise-making object such as a fan, the frequencies associated with a fan can be attenuated and removed from the entire audio signal, as opposed to only removing those frequencies when the fan is plainly audible to the second microphone or as opposed to only removing those frequencies when a particular fan-representative waveform is identified in the first or second audio signal. Comparably, if a sound of a keyboard is identified in the second audio signal, any typing sound patterns or correlated frequencies in the first audio signal can be detected and removed for the modified audio signal, whether or not the keyboard is audible to the second microphone at all times that it is audible to the first microphone.

Referring again toFIG.12, the external device1202can in some embodiments include a movement sensor1214in addition to, or instead of, a second microphone1208. The movement sensor1214can include a position or movement sensing device configured to transduce a position or movement the external device1202, such as an accelerometer, a gyroscope, an inertial measurement unit (IMU), a compass, an orientation sensor, similar devices, and combinations thereof. As the external device1202moves, the movement sensor1214can output a signal relayed to the computing device202, either directly or via the network connection206, as indicated by arrow1216.

The signal of the movement sensor1214can be used in a manner similar to the sound information described in methods4and16. For example, as shown inFIG.17, a method1700of the present disclosure can include receiving a movement signal and an audio signal, as indicated by block1702. The movement signal can be a signal provided by the movement sensor1214, and the audio signal can be provided by the microphone of the computing device202. The timing of the movement signal can be correlated to the timing of the audio signal so that detected movements from the movement sensor1214can be compared to audio signals of the microphone208.

In block1704, the method1700can include identifying a source of a noise in the movement signal. In this case, rather than detecting a source of the noise using an image recognition or sound recognition technique, the computing device can employ a movement pattern recognition technique similar to techniques employed to detect steps, running, swimming, and other activities where sensors are in motion on a user. Therefore, this method1700can beneficially be implemented in embodiments where the movement sensor1214is positioned on a wearable device (e.g.,1106,1108) that is worn by a user interacting with the computing device202. Thus, performance of block1704can include identifying movement patterns of a motion sensor on a user's arm, such as in a wristwatch, to determine the position of the user's arm relative to the computing device202and to thereby determine whether the user has their hand next to the keyboard of the computing device, whether the user is actively typing on the keyboard and thereby moving their arm in a typing manner, or making another action with their arm that indicates that they are making a noise with their arm or a portion thereof. Similarly, the performance of block1704can include identifying movement patterns of the motion sensor on a user's head, such as in a headset, headphones, visor, helmet, or other head-mounted device to determine whether the user is facing the computing device202, whether the user's mouth or jaw is moving, whether vibrations in the user's skull or jaw indicate that he or she is speaking, or other detected movements or changes in position of the user that suggests that the user is either making sound or is oriented or moving in a manner configured to avoid providing a sound to the microphone208. Thus, identifying the source of noise in the movement signal in block1704can include identifying whether a representation of the source of noise should be reduced/canceled in a modified audio signal (see block1708) or whether the representation of the source of noise should be isolated or highlighted in the modified audio signal.

In some embodiments, the computing device can play movement pattern recognition technique to detect a pattern output by a movement sensor that is part of the computing device202to determine that the computing device202is moving, such as when a user102is typing on the keyboard116, using a trackpad118, lifting the computing device202, adjusting a display120, or making other sounds with the computing device itself.

The method1700can further include identifying sound information for the source of noise identified in block1704, as indicated in block1706. In other words, the computing device can identify sound characteristics that are typical in recordings of sounds made by the source of noise identified in block1704. This can be done using the methods described above in connection with blocks406and1606. For example, if the computing device determines, via the motion sensor signals, that the source of noise is a user's hand typing on the keyboard212, typing sound information can be identified for that keyboard212or for that user's typing style so that the computing device can modify the audio signal of the microphone208to eliminate typing sounds in the audio signal in block1708.

Thus, the method1700can include modifying the audio signal using the sound information, as shown in block1708, by using the methods described above in connection with blocks408and1608. For example, the computing device can reduce or attenuate sounds in the audio signal of the microphone208that corresponds to typing sounds having the characteristics of sound information determined in block1706after detecting a characteristic movement pattern in block1704, even if the microphone208does not detect a clear, isolated typing sound in the recording made by the microphone208. Thus, by leveraging the use of multiple devices, such as computing device202and external device1202, the modified audio signal can have reduced or eliminated unwanted noises in situations where a single microphone, or even multiple microphones on a single computing device, would not be as effective.

FIG.18is a block diagram showing elements of a computing system1800that can be used in embodiments of the computing devices discloses herein (e.g., computing devices104,202and external devices902,1202). Alternatively, the computing system1800can be a separate system embodied in a remote device connectable to the computing devices disclosed herein. The computing system1800can be embodied as a personal computer, a server, a portable computing device, a set of computing devices, similar devices, and combinations thereof.

Accordingly,FIG.18is a block diagram of a computer system1800or computing device according to an embodiment of the present disclosure. In various examples, the computer system1800can include various sets and subsets of the components shown inFIG.18. Thus,FIG.18shows a variety of components that can be included in various combinations and subsets based on the operations and functions performed by the system1800in different embodiments. It is noted that, when described or recited herein, the use of the articles such as “a” or “an” is not considered to be limiting to only one, but instead is intended to mean one or more unless otherwise specifically noted herein.

The computer system1800can include a central processing unit (CPU) or processor1802connected via a bus1804for electrical communication to a memory device1806, a power source1808, an electronic storage device1810, a network interface1812, an input device adapter1816, and an output device adapter1820. For example, one or more of these components can be connected to each other via a substrate (e.g., a printed circuit board or other substrate) supporting the bus1804and other electrical connectors providing electrical communication between the components. The bus1804can include a communication mechanism for communicating information between parts of the system1800.

The processor1802can be a microprocessor, central processing unit, or a similar device configured to receive and execute a set of instructions1824stored by the memory1806. The memory1806can be referred to as main memory, such as random access memory (RAM) or another dynamic electronic storage device for storing information and instructions to be executed by the processor1802. The memory1806can also be used for storing temporary variables or other intermediate information during execution of instructions executed by the processor1802. The storage device1810can include read-only memory (ROM) or another type of static storage device coupled to the bus1804for storing static or long-term (i.e., non-dynamic) information and instructions for the processor1802. For example, the storage device1810can include a magnetic or optical disk (e.g., hard disk drive (HDD)), a solid state memory (e.g., a solid state disk (SSD)), or a comparable device. The power source1808can include a power supply capable of providing power to the processor1802and other components connected to the bus1804, such as a connection to an electrical utility grid or a battery system of an autonomous device (e.g.,100).

The instructions1824can include information for executing processes and methods using components of the system1800and other components connected to the system1800. Such processes and methods can include, for example, the methods described elsewhere herein, such as, for example, methods described in connection withFIGS.1-17.

The network interface1812can include an adapter for connecting the system1800to an external device via a wired or wireless connection. For example, the network interface1812can provide a connection to a computer network1805such as a cellular network, the Internet, a local area network (LAN), network connection206, a separate device capable of wireless communication with the network interface1812(e.g., computing device202or external devices902and1202), other external devices or network locations, and combinations thereof. In one example embodiment, the network interface1812is a wireless networking adapter configured to connect via WI-FI, BLUETOOTH®, BLUETOOTH LOW ENERGY (BLE), long-term evolution (LTE), 5G, a mesh network, or a related wireless communications protocol to another device having interface capability using the same protocol. In some embodiments, a network device or set of network devices in the network1805can be considered part of the system1800. In some examples, a network device can be considered connected to, but not a part of, the system1800.

The input device adapter1816can be configured to provide the system1800with connectivity to various input devices such as, for example, a computer input device1814(e.g., keyboard116or212or mouse118), cameras1815(e.g.,122,210,802, or904), microphones1817(e.g.,208or1208), movement sensors1819(e.g.,1214), one or more other sensors, related devices, and combinations thereof.

The output device adapter1820can be configured to provide the system1800with the ability to output information to a user, such as by providing visual output using one or more displays1832and by providing audible output using one or more speakers1835. The processor1802can be configured to control the output device adapter1820to provide information to a user via the output devices connected to the adapter1820.

The instructions1824can include electronic instructions that, when executed by the processor1802, can perform methods and processes as described in further detail elsewhere herein. The instructions1824can be stored or encoded on a non-transitory computer readable medium, and the instructions1824, when executed by a computing device such as, for example, processor1802, cause the computing device to perform methods and processes as described in further detail elsewhere herein. See, e.g.,FIGS.4,16, and17.

To the extent applicable to the present technology, gathering and use of data available from various sources can be used to improve the delivery to users of invitational content or any other content that may be of interest to them. The present disclosure contemplates that in some instances, this gathered data may include personal information data that uniquely identifies or can be used to contact or locate a specific person. Such personal information data can include demographic data, location-based data, telephone numbers, email addresses, TWITTER® ID's, home addresses, data or records relating to a user's health or level of fitness (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other identifying or personal information.