Patent Description:
This disclosure relates generally to the field of communication systems, and, more particularly to establishing a communication link with a remote internet of things (IoT) device of a communication network.

Internet of things (IoT) can refer to network enablement of devices that typically perform some other non-communications function. The IoT devices are embedded with electronics, software, sensors, actuators, and network connectivity components to enable the IoT devices to collect and exchange data and also to be sensed or controlled remotely across existing network infrastructure. For example, users can use devices, such as mobile phones, to run various types of applications that access and control the IoT devices directly or through the network infrastructure to collect data or perform a function.

A user can use a camera application and the related camera components that are available in various types of devices (such as mobile phones, dedicated cameras, tablet computers, drones, or smart glasses) to record audio and video (A/V). When a user uses a device having a camera to record audio and video (A/V) of a subject positioned in the field of view (FoV) of the camera, the video is captured using the camera components of the device and the audio is recorded using the microphone of the same device that is recording the video. However, when the subject being recorded is not at close proximity to the camera (such as being located at a distance greater than five meters away) and is located in a noisy environment, the recorded audio may have a substantial amount of ambient noise. In some cases, the ambient noise may substantially interfere with the target audio from the subject being recorded. For example, the recorded audio of a presenter in a noisy auditorium may have a substantial amount of ambient noise because the microphone is at a distance from the presenter and the environment is noisy. In another example, when recording A/V of a person giving a toast at a wedding or party, the noise in the environment may completely drown out the target audio from the person giving the toast.

<CIT> discloses an apparatus comprising: an audio source determiner configured to determine at least one audio source; a visualizer configured to generate a visual representation associated with the at least one audio source; and a controller configured to process an audio signal associated with the at least one audio source dependent on interaction with the visual representation.

<CIT> discloses an apparatus comprising: an input configured to receive at least one audio signal from a further apparatus; an input configured to receive at least one audio signal associated with the apparatus; an orientation/location determiner configured to determine a relative orientation/location difference between the apparatus and the further apparatus; an audio processor configured to process the at least one audio signal from the further apparatus based on the relative orientation/location difference between the apparatus and the further apparatus; and a combiner configured to combine the at least one audio signal from the further apparatus having been processed and the at least one audio signal associated with the apparatus.

<CIT> discloses an information collection system including an information collection apparatus, a photographing apparatus and information acquisition apparatuses. The information collection apparatus collects information on the information acquisition apparatuses and the photographing apparatus. The information collection apparatus selects at least one of the information-acquisition apparatuses based on the collected information, and collects information about a predetermined position by communicating with the selected information-acquisition apparatus.

The systems, methods, and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

One innovative aspect of the subject matter described in this disclosure can be implemented by a computing device for network communication for recording audio and video (A/V). The computing device may determine a plurality of internet of things (IoT) devices that are displayed within a field of view (FoV) of a camera of the computing device and that are capable of remote audio recording and audio streaming. The computing device may select a first loT device from the plurality of IoT devices. The first loT device may have a first microphone that is capable of remotely recording audio of a subject for the computing device. The computing device may receive an audio stream from the first loT device after selecting the first loT device. The audio stream may include the audio of the subject remotely recorded by the first loT device for the computing device.

In some implementations, the computing device may determine, within the FoV of the camera of the computing device, distance measurements from each of the plurality of IoT devices to a focal point of the FoV. The computing device may select the first IoT device from the plurality of loT devices based, at least in part, on the distance measurements.

In some implementations, the computing device may determine, based on the distance measurements, that the first loT device from the plurality of loT devices is located closest to the focal point of the FoV. The computing device may select the first loT device from the plurality of loT devices based, at least in part, on determining that the first loT device is located closest to the focal point of the FoV.

In some implementations, the computing device may determine a microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices. The computing device may compare the microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices. The computing device may select the first IoT device from the plurality of IoT devices based, at least in part, on the microphone type and performance characteristics associated with the first microphone of the first IoT device.

In some implementations, the computing device may determine that the first IoT device and a second IoT device of the plurality of IoT devices are displayed within a focal area of the FoV of the camera of the computing device. The computing device may receive a first audio stream from the first IoT device and a second audio stream from the second IoT device. The computing device may compare audio quality characteristics of the first audio stream to audio quality characteristics of the second audio stream. The computing device may select the first IoT device based, at least in part, on the audio quality characteristics of the first audio stream received from the first IoT device.

In some implementations, the computing device may receive user input selecting the first IoT device having the first microphone. The computing device may select the first IoT device from the plurality of IoT devices based, at least in part, on the user input.

In some implementations, the computing device may record video of the subject using the camera of the computing device, and generate an A/V recording of the subject using the video of the subject recorded by the camera of the computing device and the audio of the subject remotely recorded by the first microphone of the first IoT device.

In some implementations, the computing device may detect a change in the FoV of the camera of the computing device. The computing device may determine to switch from receiving the audio stream from the first IoT device to receiving an audio stream from a second IoT device after detecting the change in the FoV of the camera. The second IoT device may have a second microphone that is capable of remotely recording the audio of the subject.

In some implementations, the computing device may determine that a second IoT device is located closer to a focal point of the FoV than the first IoT device after detecting a change in the FoV of the camera. The computing device may determine to switch from receiving the audio stream from the first IoT device to receiving the audio stream from the second IoT device in response to determining that the second IoT device is located closer to the focal point of the FoV than the first IoT device after detecting the change in the FoV of the camera.

Another innovative aspect of the subject matter described in this disclosure can be implemented in a computing device comprising a processor and memory having instructions stored therein which, when executed by the processor, cause the computing device to determine a plurality of IoT devices that are displayed within a FoV of a camera of the computing device and that are capable of remote audio recording and audio streaming, select a first IoT device from the plurality of IoT devices, where the first IoT device may have a first microphone that is capable of remotely recording audio of a subject for the computing device, and receive an audio stream from the first IoT device after selection of the first IoT device, where the audio stream may include the audio of the subject remotely recorded by the first IoT device for the computing device.

Another innovative aspect of the subject matter described in this disclosure can be implemented in a computer-readable storage medium having instructions stored therein, which when executed by a processor of a computing device, cause the computing device to determine a plurality of IoT devices that are displayed within a FoV of a camera of the computing device and that are capable of remote audio recording and audio streaming, select a first IoT device from the plurality of IoT devices, where the first IoT device may have a first microphone that is capable of remotely recording audio of a subject for the computing device, and receive an audio stream from the first IoT device after selection of the first IoT device, where the audio stream may include the audio of the subject remotely recorded by the first IoT device for the computing device.

Another innovative aspect of the subject matter described in this disclosure can be implemented by a computing device for network communication. The computing device may detect a second computing device that is located within a FoV of a camera of the first computing device. The computing device may provide a request message from the first computing device to the second computing device in response to detecting the second computing device within the FoV of the camera of the first computing device. The request message may request approval to establish a communication link between the first computing device and the second computing device. The computing device may receive a response message from the second computing device. The response message may grant approval to establish the communication link between the first computing device and the second computing device. The computing device may establish the communication link between the first computing device and the second computing device for the first computing device to communicate with the second computing device.

The following description is directed to certain implementations for the purposes of describing the innovative aspects of this disclosure. However, a person having ordinary skill in the art will readily recognize that the teachings herein can be applied in a multitude of different ways. The described implementations may be implemented in any device, system or network that is capable of transmitting and receiving radio frequency (RF) signals according to any of the Institute of Electrical and Electronics Engineers (IEEE) <NUM> standards, or any of the IEEE <NUM> standards, the Bluetooth® standard, code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), Global System for Mobile communications (GSM), GSM/General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Terrestrial Trunked Radio (TETRA), Wideband-CDMA (W-CDMA), Evolution Data Optimized (EV-DO), 1xEV-DO, EV-DO Rev A, EV-DO Rev B, High Speed Packet Access (HSPA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Evolved High Speed Packet Access (HSPA+), Long Term Evolution (LTE), AMPS, or other known signals that are used to communicate within a wireless, cellular or internet of things (IoT) network, such as a system utilizing <NUM>, <NUM> or <NUM>, or further implementations thereof, technology.

With the proliferation of IoT technology, many devices in homes, businesses, event venues, and other locations may include IoT devices (or will soon be upgraded with IoT devices) that are networked together and connected to the existing network infrastructure, such as through a local area network (LAN) and to the Internet. Many of the IoT devices have microphones that can support voice commands and record audio. Example IoT devices that typically have microphones include smart appliances, smart lighting components (such as smart switches, lighting fixtures, and light bulbs), smart thermostats, smart security system components (such as smart control panels, cameras, and sensors), smart media devices (such as smart televisions, home assistants, projectors, and speakers), smart door locks, dedicated sensors, and the like. Computer-based devices (also referred to herein as "computing devices") can be used to communicate with the IoT devices to remotely access the IoT devices, exchange commands and information with the IoT devices, and control the IoT devices to improve, simplify, and automate day-to-day tasks.

A camera of a computing device (such as a dedicated camera, a mobile phone, a tablet computer, a drone, smart or augmented reality (AR) glasses, or the like) may be used to record audio and video (A/V) of a subject in the field of view (FoV) of the camera. In some instances, the subject being recorded may not be in close proximity to the camera of the computing device (such as being located at a distance greater than a few meters away, like three, four, five or more meters away) and the subject is located in a noisy environment. For example, the subject being recorded may be a presenter in a noisy auditorium or a person giving a toast at a wedding or party. In some implementations, the computing device may obtain the audio for the A/V recording from one or more microphones of one or more remote IoT devices that are located at close proximity to the subject being recorded. For example, a microphone of a remote IoT device may record audio of the user and stream the audio to the computing device. The computing device may capture the video for the A/V recording using the camera of the computing device. Thus, the computing device may record the video locally using the camera components of the computing device, and obtain the audio remotely from one or more microphones of one or more IoT devices of the local network to generate the A/V recording.

When the camera application of the computing device is accessed for recording A/V of the subject, the computing device may display visual indicators (such as icons or markers) in the FoV of the camera showing the location of the IoT devices that have a microphone and that are capable of remotely recording and streaming audio. For example, the computing device may obtain a map of the IoT devices of a local network from a master device of the local network. The computing device may use the map to determine the location of the IoT devices in order to display the visual indicators in the FoV of the camera.

In some implementations, the computing device may select a single IoT device having a microphone that is capable of remote recording and streaming audio for the A/V recording. For example, the computing device may select the IoT device that is closest to the subject being recorded. In another example, the computing device may select the IoT device that has a preferred microphone type and performance characteristics (or specifications). In another example, the computing device may select the IoT device that is streaming the best quality audio to the computing device. In some implementations, the computing device may select two or more IoT devices to remotely record and stream the audio. The two or more IoT devices also may be selected based on at least one of the proximity to the subject being recorded, the microphone type and performance characteristics of the microphones, and the quality of the streamed audio that is received at the computing device. In some implementations, the two or more IoT devices may be selected based on their proximity to one another and to the subject being recorded. In some implementations, the user of the camera also may override the automatic selection of the one or more IoT devices by the computing device, and instead use the display of the computing device to select the one or more IoT devices. Also, the computing device may designate the local microphone of the computing device as the backup microphone in case the communication link with the one or more IoT devices is lost or other issues affect the remote recording and streaming of the audio.

After selecting the one or more IoT devices, the computing device may begin receiving the audio stream from the one or more IoT devices during the A/V recording. In some implementations, when the FoV of the camera changes during the A/V recording, such as by the user moving the camera or the user zooming in or out, the computing device can dynamically switch from receiving the audio stream from a first IoT device to receiving the audio stream from a second IoT device.

In some implementations, a first computing device of a first user may detect a second computing device of a second user that is displayed within a FoV of a camera of the first computing device. For example, the first user wearing a pair of smart glasses looks in the direction of the second user that is also wearing a pair of smart glasses. The first user's smart glasses may detect the second user's smart glasses in the FoV of the camera of the first user's smart glasses. The first computing device may provide a request message to the second computing device to request approval to establish a communication link between the first computing device and the second computing device for the first user to communication with the second user from a distance.

Particular implementations of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages. The computing device can generate an A/V recording having good quality audio with reduced ambient noise even when recording A/V from a distance in a noisy environment. The remote recording and streaming of audio from one or more remote IoT devices during the A/V recording of a subject from a distance utilizes the networked IoT environment to greatly improve the audio quality of the A/V recording. The computing device can use the networked IoT environment and the FoV of the computing device's camera to dynamically select and switch between IoT devices to improve the audio quality. The computing device may use two or more IoT devices of the networked IoT environment to improve the audio quality by using noise cancellation. Also, the selection of the one or more IoT devices and the remote recording and streaming of the audio is automatic and seamless, which maintains the ease of use for the user. Furthermore, a first computing device may dynamically and seamlessly connect to a second computing device based on the FoV of the camera of the first computing device to allow the first user to communicate with the second user from a distance.

<FIG> shows an example diagram of a computing device displaying an IoT environment and selecting one of the IoT devices for remotely recording and streaming audio for an A/V recording of a subject. The IoT environment may be any indoor or outdoor space in a home, office, commercial building, government building, public park, event venues, and the like, that has IoT devices as part of the network infrastructure. Examples of indoor or outdoor spaces that may have IoT devices include auditoriums, living rooms, backyards, conference rooms, stages, lobbies, classroom, courtyards, amphitheaters, and the like. The IoT environment <NUM> shown in <FIG> is an auditorium that includes IoT devices 120A, 120B, 120C, 120D, and 120E (which may collectively be referred to as IoT devices <NUM>). In the example shown in <FIG>, IoT device 120A is a smart lighting fixture or a smart light bulb, IoT device 120B is a smart light switch, IoT device 120C is a smart media device (such as a smart projector), IoT device 120D is a home/office assistant or a master device of a local wireless network (such as a wireless LAN access point (AP)), and IoT device 120E is a smart power plug. It is noted that the IoT environment may include additional IoT devices not shown in <FIG> for simplicity.

A computing device <NUM> may be used to record A/V of a subject <NUM> in the IoT environment <NUM> from a distance. In the example shown in <FIG>, the computing device <NUM> is a mobile phone and the subject <NUM> being recorded is a presenter in an auditorium. The computing device <NUM> may be various types of devices that have a camera, such as a mobile phone, a dedicated camera, a tablet computer, a drone, smart glasses, or the like. Smart glasses may include augmented reality (AR) glasses, virtual reality (VR) glasses, or mixed reality (MR) glasses that implement aspects of both AR and VR. To record A/V of the subject <NUM>, the user of the computing device <NUM> may open the camera application using a touchscreen (or physical buttons) of the computing device <NUM> or using a voice command. Then, the user may point the camera of the computing device <NUM> in the direction of the subject <NUM> in order to display the subject <NUM> and the IoT environment <NUM> in the FoV <NUM> of the camera of the computing device <NUM>, as shown in <FIG>. For example, the user can place the subject <NUM> near the center of the focus area <NUM> of the FoV <NUM>. In one example, when using a mobile phone, the FoV <NUM> of the camera can be displayed to the user in the main display of the mobile phone, and the user can move the mobile phone to place the subject <NUM> near the center of the focus area <NUM>. In another example, when using smart glasses, the FoV <NUM> of the camera can be displayed to the user in one or more displays that may be located in one or more lenses of the smart glasses, and the user can look in the direction of the subject <NUM> to place the subject <NUM> near the center of the focus area <NUM>. The center of the focus area <NUM> also may be referred to as the focal point <NUM> of the FoV <NUM>. Some cameras display the focus area <NUM> of the FoV <NUM> as a rectangle or square with dashed lines, and the focal point <NUM> with a cross or a dot at the center of the focus area <NUM>. Some cameras display either the focus area <NUM> or the focal point <NUM>, but not both. Users typically position the subject <NUM> being recorded within the focus area <NUM> because the camera's autofocus and other image processing features optimize the focus and improves the overall quality of the portions of the video or image being captured within the focus area <NUM>.

In some implementations, the computing device <NUM> may determine the general location of the IoT devices <NUM> and determine whether each of the IoT devices <NUM> supports remote recording and streaming of audio. As shown in <FIG>, the camera of the computing device <NUM> displays (or overlays) visual indicators 130A, 130B, 130C, and 130D (which may collectively be referred to as visual indicators <NUM>) within the FoV <NUM> of the camera. The visual indicators <NUM> show the general location of the IoT devices <NUM> that have a microphone and that support remote recording and streaming of audio in the IoT environment <NUM>. For example, the IoT device 120A is displayed with a visual indicator 130A, the IoT device 120B is displayed with a visual indicator 130B, the IoT device 120C is displayed with a visual indicator 130C, and the IoT device 120D is displayed with a visual indicator 130D. The IoT device 120E is not displayed in the FoV <NUM> with a visual indicator because the computing device <NUM> determined that the IoT device 120E doesn't have a microphone or the microphone it has does not support remote recording and streaming of audio.

In some implementations, the computing device <NUM> may communicate with a master device (such as IoT device 120D) of the local network to obtain a map of the IoT devices <NUM> of the IoT environment <NUM>. For example, the master device may be a WLAN access point, a local network server, a smart assistant device, or the like. The master device may either store the map of the IoT devices <NUM> locally or in a remote database accessed via the local network or the Internet. In one example, the computing device <NUM> may obtain an Indoor Positioning System (IPS) map from the master device that shows the locations of the IoT devices <NUM> within the IoT environment <NUM>. The IPS map may be predetermined by a network administrator using various indoor positioning techniques that are known in the art. The IPS map may be stored locally at the master device or remotely in a database. The computing device <NUM> can merge (or overlay) the map of the IoT devices <NUM> with the camera's FoV <NUM> to determine the location of the IoT devices <NUM>.

In some implementations, the map of the IoT devices <NUM> obtained from the master device shows the location of all available IoT devices <NUM> of the IoT environment <NUM>, including IoT devices that do not have microphones. Thus, in one example, the computing device <NUM> communicates with the master device to request information about the IoT devices <NUM>, such as to request at least one of the type of IoT device, the operational capabilities of each IoT device, and the specifications of each IoT device. In another example, the computing device <NUM> can establish a communication link with each of the IoT devices <NUM> and request the above-referenced information directly from each IoT device. Based on the information received from the master device or directly from the IoT devices <NUM>, the computing device <NUM> can identify which of the IoT devices <NUM> have a microphone and support remote recording and streaming of audio. The computing device <NUM> can then display (or overlay) the visual indicators <NUM> within the FoV <NUM> of the camera to show the general location of the IoT devices <NUM> within the IoT environment <NUM> that have a microphone and that support remote recording and streaming of audio, as shown in <FIG>.

In some implementations, the computing device <NUM> may select one of the IoT devices <NUM> having a microphone and that is capable of remotely recording and streaming audio for the A/V recording of the subject <NUM>. For example, the computing device <NUM> selects one of the IoT devices <NUM> that are displayed with a visual indicator <NUM>. The visual indicator <NUM> indicates that the IoT device has a microphone and supports remotely recording and streaming audio. If only one of the IoT devices <NUM> that are shown within the FoV <NUM> is displayed with a visual indicator <NUM>, the computing device <NUM> dynamically selects the IoT device for the A/V recording of the subject <NUM>. When two or more IoT devices <NUM> are displayed with a visual indicator <NUM> (as shown in <FIG>), the computing device <NUM> may select one of the IoT devices. In one example, the computing device <NUM> may select the IoT device that is closest to focal point <NUM> of the FoV <NUM>, since users typically place the subject <NUM> close to the focal point <NUM> of the FoV <NUM> when recording video of the subject <NUM>. For example, the IoT device 120A and the IoT device 120B of <FIG> are shown within the focus area <NUM> and are the closest IoT devices <NUM> to the focal point <NUM> of the FoV <NUM> (and also to the subject <NUM> being recorded). In this example, the computing device <NUM> may select the IoT device 120A because it is closer to the focal point <NUM> than the IoT device 120B. In another example, the computing device <NUM> may compare the microphone types and performance characteristics (or specifications) of the IoT devices <NUM>, or may compare the quality of the audio received from the IoT devices <NUM>. The computing device <NUM> may select one of the IoT devices <NUM> with a preferred microphone type and performance characteristics (or specifications), or may select the IoT device <NUM> that streamed the best quality audio, as will be further described with reference to <FIG>.

The user of the computing device <NUM> also can manually select, using the touchscreen of the computing device <NUM> or by a voice command, one of the IoT devices <NUM> (such as IoT device 120A) for the A/V recording of the subject <NUM>. The computing device <NUM> also may detect other types of commands, such as commands based on the user's eye movement and eye blinks or other types of biometric commands. The user also may override the automatic selection of one of the IoT device <NUM> by the computing device <NUM> and use the touchscreen of the computing device <NUM> (or a voice command) to manually select a different IoT device (such as IoT device 120B). When the computing device <NUM> (or the user) selects one of the IoT devices <NUM> for the A/V recording, the computing device <NUM> designates the local microphone of the computing device <NUM> as a backup microphone in case the communication link with the selected IoT device is lost or there are other network issues. After selecting one of the IoT devices <NUM>, the computing device <NUM> communicates with the selected IoT device and begins receiving the audio stream from the selected IoT device during the A/V recording, as will be further described in <FIG> and <FIG>.

In some implementations, after the user starts the camera application, the computing device <NUM> may initially establish a communication link with each of the IoT devices <NUM> that are displayed with the FoV <NUM> of the camera or with only the IoT devices <NUM> that are displayed within the focus area <NUM> of the FoV <NUM>. The microphones of the IoT devices <NUM> that establish a communication link with the computing device <NUM> may be dynamically turned on when communication link is established. The microphones of the IoT devices <NUM> that do not establish a communication link with the computing device <NUM> may remain in an off or low power state. Also, after the computing device <NUM> selects one or more of the IoT devices <NUM> for the remote audio recording and streaming, the microphones of the IoT devices <NUM> that are not selected may be turned off or may switch to a low power state. Dynamically switching the microphones on and off depending on whether they are being used by the computing device <NUM> can reduce power consumption, and also may help extend the life of the batteries of the IoT devices <NUM> that are battery operated.

<FIG> shows a block diagram of an example computing device and an example IoT device. The computing device <NUM> may include a communication unit <NUM>, a camera module <NUM>, a microphone module <NUM>, an A/V processing unit <NUM>, and user interface <NUM>. The IoT device <NUM> may be representative of any of the IoT devices <NUM> of <FIG>, such as IoT device 120A. The IoT device <NUM> may include a communication unit <NUM> and a microphone module <NUM>. The communication units <NUM> and <NUM> may each include an analog front end (AFE), one or more antennas, one or more modems (such as a Bluetooth modem and an <NUM> modem), and other related communication components. The camera module <NUM> may include the camera lenses, flash, and other related camera components. The microphone modules <NUM> and <NUM> may each include one or more microphones and other related microphone components. The A/V processing unit <NUM> may include one or more processor cores, cache, memory, and other related components. In some implementations, the A/V processing unit <NUM> may be part of the mobile processor(s) or the graphic processor(s) of the computing device <NUM>. The user interface <NUM> may include the display, touchscreen, speakers, physical buttons, and other related user interface components. The camera module <NUM>, the microphone module <NUM>, and other related sensors also may be part of the user interface components of the computing device <NUM> in order to detect voice commands, eye movement commands, and other types of biometric commands. It is noted that the computing device <NUM> and the IoT device <NUM> may have additional components which are not shown in <FIG> for simplicity.

After the computing device <NUM> selects one of the IoT devices <NUM> (such as IoT device 120A), the computing device <NUM> communicates with the selected IoT device <NUM>. For example, the computing device <NUM> may send a request message with a command <NUM> (or other type of request message) to the selected IoT device <NUM> using the communication unit <NUM>, the command requesting the selected IoT device <NUM> to initiate the remote recording and streaming of the audio. The selected IoT device <NUM> may optionally send an acknowledge message to the computing device <NUM>. The selected IoT device <NUM> may then begin recording audio using the microphone module <NUM>, and also transmitting an audio stream <NUM> to the computing device <NUM> using the communication unit <NUM>. The computing device <NUM> may receive the audio stream <NUM> using the communication unit <NUM> and provide the audio to the A/V processing unit <NUM> for further processing. The computing device <NUM> also may provide the video recorded using the camera module <NUM> to the A/V processing unit <NUM> for further processing. When the local microphone of the computing device <NUM> is recording the audio, the microphone module <NUM> provides the audio to the A/V processing unit <NUM>. The A/V processing unit <NUM> merges the recorded audio and video to generate the A/V recording. The A/V processing unit <NUM> may store the A/V recording and also provide the A/V recording to components of the user interface <NUM>, such as the display and speakers, to present the A/V recording to the user.

As described in <FIG>, during the A/V recording of the subject <NUM>, when the FoV <NUM> of the camera changes, such as by the user moving the computing device <NUM> or the user zooming in or out, the computing device <NUM> can switch from receiving the audio stream from a first IoT device (such as IoT device 120A) to receiving the audio stream from a second IoT device (such as IoT device 120B).

<FIG> shows another example diagram of the computing device displaying the IoT environment and switching from receiving an audio stream from a first IoT device to receiving an audio stream from a second IoT device after the FoV of the camera of the computing device is changed. The diagram of <FIG> shows the same IoT environment <NUM> and the same subject <NUM> as the diagram of <FIG> (the presenter in the auditorium), except that the FoV <NUM> of the camera of the computing device <NUM> has been shifted to the left in <FIG> (from the perspective of the computing device <NUM>) when the subject <NUM> moved to the left toward the presentation screen. For example, when the subject <NUM> moved left toward the presentation screen, the user moved the computing device <NUM> to the left in order to keep the subject <NUM> near the focal point <NUM> of the focus area <NUM> of the FoV <NUM>. When the FoV <NUM> was shifted to the left, the IoT device 120E (a smart power plug shown in <FIG>) is now outside the FoV <NUM> shown in <FIG>, and IoT devices 320F and <NUM> are now displayed within the FoV <NUM> shown in <FIG>. The IoT device 320F may be another smart light switch, and the IoT device <NUM> may be another smart power plug.

The FoV <NUM> of the camera of the computing device <NUM> is changed when the user moves the computing device <NUM> in any direction, such as left, right, up, down, forward, backward, or some combination of the directions. The FoV <NUM> of the camera also can be changed when the user zooms in or zooms out by using the zoom feature that is available in most cameras. Cameras may have various other features that can change at least one of the FoV <NUM>, the focus area <NUM>, and the focal point <NUM>. For example, in some cameras, the user can move the focus area <NUM> and the focal point <NUM> to different areas within the FoV <NUM>.

In <FIG>, the computing device <NUM> may initially select the IoT device 120A to remotely record and stream audio for the A/V recording of the subject <NUM>. In some implementations, when the FoV <NUM> of the camera changes, the computing device <NUM> may switch from receiving an audio stream from first IoT device (such as the IoT device 120A) to receiving an audio stream from a second IoT device. For example, the computing device <NUM> may determine which of the IoT devices <NUM> is closest to the focal point <NUM> (which may correspond to the location of the subject <NUM>) when the FoV <NUM> of the camera changes, and then switch to receiving the audio stream from the second IoT device if the second IoT device is closer to the focal point <NUM> than the first IoT device. In the example shown in <FIG>, when the subject <NUM> moved to the left and the user shifted the FoV <NUM> of the camera by moving the computing device <NUM> to the left, the focal point <NUM> of the FoV <NUM> shifted closer to the IoT device 120B than to the IoT device 120A. Thus, when the computing device <NUM> detects a change in the FoV <NUM>, the computing device <NUM> may determine the distance between each of the IoT devices <NUM> and the focal point <NUM>. In this example, the computing device <NUM> determines that the IoT device 120B is now closer to the focal point <NUM> than the IoT device 120A. The computing device <NUM> then automatically and seamlessly switches to receiving the audio stream from the IoT device 120B, as will be further described with reference to <FIG>. Also, while receiving the audio stream from one of the IoT devices <NUM> and while switching from one IoT device to another IoT device, the computing device <NUM> may designate the local microphone of the computing device <NUM> as the backup microphone in case there are technical issues with the communication link or in case there are issues with the switch from one IoT device to another IoT device.

<FIG> shows an example message flow diagram of a computing device selecting one of the IoT devices for remotely recording and streaming audio during the A/V recording of the subject. The message flow diagram <NUM> includes messages between the computing device <NUM>, a master device <NUM>, a first IoT device 420A, and a second IoT device 420B. It is noted that the master device <NUM>, the first IoT device 420A, and the second IoT device 420B may correspond to the IoT device 120D, the IoT device 120A, and the IoT device 120B of <FIG>, respectively. However, the master device <NUM>, the first IoT device 420A, and the second IoT device 420B also may be different devices in the IoT environment <NUM> or may be devices in a different IoT environment not shown.

At <NUM>, the computing device <NUM> starts the camera application in response to receiving input from the user. The FoV <NUM> of the camera displays the IoT environment and the subject that will be recorded. At <NUM>, the computing device <NUM> sends a request message to the master device <NUM> of the local network to request a map of the IoT devices of the IoT environment. The request message also may request additional information regarding the IoT devices, such as at least one of the type of IoT device, the operational capabilities of each IoT device, and the specifications of each IoT device. The master device <NUM> may either store the map of the IoT devices (and the other information about the IoT devices) locally or in a remote database accessed via the local network or the Internet. At <NUM>, the master device <NUM> sends a response message to the computing device <NUM> that includes the map of the IoT devices. The map shows the location of the IoT devices within the IoT environment, including the location of the first IoT device 420A and of the second IoT device 420B. The response message also may include the additional information regarding the IoT devices that was requested by the computing device <NUM>. If the master device <NUM> does not have additional information regarding the IoT devices, the master device <NUM> only sends the map of the IoT devices to the computing device <NUM>. If the master device <NUM> does not have a map of the IoT devices, the request message may indicate that the master device does not have map of the IoT devices.

At <NUM>, the computing device <NUM> merges the map of the IoT devices with the camera's FoV <NUM> to determine the location of the IoT devices. When the computing device <NUM> receives the requested information regarding the IoT devices (such as the type of IoT device, the operational capabilities of each IoT device, or the specifications of each IoT device) from the master device <NUM>, the computing device <NUM> can determine that the first IoT device 420A and the second IoT device 420B have microphones and support remotely recording and streaming audio. When the computing device <NUM> merges the map of the IoT devices with the camera's FoV <NUM>, the computing device <NUM> adds a visual indication (such as an icon) to the FoV <NUM> showing the location of the first IoT device 420A and the second IoT device 420B within the IoT environment and indicating that the first IoT device 420A and the second IoT device 420B support remote recording and streaming of audio. If the master device <NUM> does not provide the requested information regarding the IoT devices, at <NUM>, the computing device <NUM> optionally sends a request message to both the first IoT device 420A and the second IoT device 420B requesting this information. The first IoT device 420A and the second IoT device 420B receive the request message and send back a response message that includes the requested information. The operational capabilities and specifications of each IoT device also may include the characteristics and specifications of each device's microphone. The request messages that are sent directly to each IoT device are optional because in some cases the master device <NUM> may have already provided the requested information.

At <NUM>, the computing device <NUM> selects either the first IoT device 420A or the second IoT device 420B to remotely record and stream audio for the A/V recording of the subject. In some implementations, the computing device <NUM> may select the IoT device that is closest to the focal point <NUM> of the FoV <NUM> of the camera (which typically corresponds to the location of the subject being recorded). In the example of <FIG>, the computing device <NUM> determines the distance between the first IoT device 420A and the focal point <NUM> of the FoV <NUM>, and also determines the distances between the second IoT device 420B and the focal point <NUM> of the FoV <NUM>. In some implementations, the computing device <NUM> may perform some measurements using a grid, and designate the focal point <NUM> of the FoV <NUM> as the origin in the X, Y axis. Thus, the focal point has the X, Y coordinates of (<NUM>, <NUM>). The computing device <NUM> may determine that the first IoT device 420A has X, Y coordinates of (<NUM>, <NUM>), and the second IoT device 420B has X, Y coordinates of (-<NUM>, <NUM>). Thus, in this example, the first IoT device 420A is closer to the focal point than the second IoT device 420B. If the computing device <NUM> determines that the first IoT device 420A is closer to the focal point than the second IoT device 402B, the computing device <NUM> select the first IoT device 420A to remotely record and stream the audio for the A/V recording. In some implementations, the computing device <NUM> selects the IoT device that has a preferred microphone type and performance characteristics (or specifications). For example, the computing device <NUM> may use Table <NUM> to determine whether one of the IoT devices has a preferred microphone type with the best performance characteristics. For illustration purposes, Table <NUM> lists three different microphone types (Ribbon, Dynamic, and Condenser) and also the ratings (and the corresponding value assigned to each rating) that were determined for the output, ruggedness, sound quality, and throw associated with the microphones.

Based on the values assigned to each rating, and the average of the values associated with each microphone type, the computing device <NUM> may determine that the Dynamic microphone type is the preferred microphone type. If the computing device <NUM> determines that the first IoT device 420A has the Dynamic microphone type and the second IoT device 420B has the Ribbon microphone type, the computing device <NUM> may select the first IoT device 420A. If both the first IoT device 420A and the second IoT device 420B have the same microphone type (such as the Dynamic microphone type), the computing device <NUM> may compare additional performance characteristics and specifications of the microphones to determine which IoT device to select for the A/V recording. For example, the computing device <NUM> may compare the sensitivity, overload, distortion, noise, and other parameters associated with the microphones to determine which IoT device to select. If the computing device <NUM> determines that the microphones are of the same quality, the computing device <NUM> can select the IoT device that is closest to the focal point <NUM> of the FoV <NUM>.

In some implementations, the computing device <NUM> may receive an audio stream from each of the first IoT device 420A and the second IoT device 420B, and the computing device <NUM> may process the audio streams to select the IoT device that streamed the best quality audio. For example, the first IoT device 420A and the second IoT device 420B may each send an audio stream to the computing device <NUM> in response to receiving, at <NUM>, the request message from the computing device <NUM>. In another example, the computing device <NUM> may establish a communication link with each of the first IoT device 420A and the second IoT device 420B when the computing device <NUM> communicates with the master device, at <NUM>. The first IoT device 420A and the second IoT device 420B may begin streaming the audio after the communication link is established between the IoT devices and the computing device <NUM>. After receiving the audio streams, the computing device <NUM> may sample each audio stream and process the samples from each audio stream to determine the best quality audio stream. For example, the computing device may compare various audio quality characteristics, such as the sound quality, distortion, noise, and other characteristics or parameters to determine the best quality audio stream. The computing device <NUM> may select the first IoT device 420A based on the audio quality characteristics of the audio stream received from the first IoT device 420A.

At <NUM>, the computing device <NUM> sends a message to the first IoT device 420A with a command to begin recording audio and streaming the audio to the computing device <NUM>. At <NUM>, the first IoT device 420A processes the command received from the computing device <NUM> and, at <NUM>, optionally sends back an acknowledgement message. At <NUM>, the first IoT device 420A begins recording and streaming the audio to the computing device <NUM>. At <NUM>, the computing device <NUM> receives the audio stream from the first IoT device 420A and merges the audio with the recorded video to generate the A/V recording of the subject.

<FIG> shows an example message flow diagram of a computing device switching from receiving an audio stream from a first IoT device to receiving an audio stream from a second IoT device after the FoV of the camera of the computing device is changed. Similar to <FIG>, the message flow diagram <NUM> of <FIG> includes messages between the computing device <NUM>, the first IoT device 420A, and the second IoT device 420B.

At <NUM>, the first IoT device 420A continues to record and stream the audio for the computing device <NUM>, as described in <FIG>. In <FIG>, the audio stream that is received by the computing device <NUM> from the first IoT device 420A may be referred to as Audio Stream <NUM>. At <NUM>, the computing device <NUM> continues to process the Audio Stream <NUM>, and at the same time monitors the FoV <NUM> of the camera of the computing device <NUM>. At <NUM>, the computing device <NUM> determines that the FoV <NUM> of the camera of the computing device <NUM> has changed. For example, the computing device <NUM> may determine that the user moved the computing device <NUM>, or that the user zoomed in or out using the zoom camera features. When the computing device <NUM> detects that the FoV of the camera has changed, the computing device <NUM> may determine to change from receiving the Audio Stream <NUM> from the first IoT device 420A to receiving an audio stream (such as Audio Stream <NUM>) from the second IoT device 420B. The computing device <NUM> may determine to switch to receiving the audio stream from a different IoT device for various reasons, such as determining that the second IoT device 420B is now closer to the focal point of the FoV than the first IoT device 420A after the FoV of the camera changed. At <NUM>, the computing device <NUM> sends a message to the second IoT device 420B with a command to begin remotely recording audio and streaming the audio to the computing device <NUM>.

In some implementations, the computing device <NUM> may coordinate the sending of commands to the IoT devices to start and stop the recording and streaming of audio in order to seamlessly switch from receiving the Audio Stream <NUM> from the first IoT device 420A to receiving the Audio Stream <NUM> from the second IoT device 420B. For example, when the computing device <NUM>, the first IoT device 420A, and the second IoT device 420B communicate using protocols from the IEEE <NUM> standards (such as <NUM> b/g/n/ac/ax), the computing device <NUM> may send a command to the second IoT device 420B to begin remotely recording and streaming the audio, while still receiving the Audio Stream <NUM> from the first IoT device 420A. In other words, since the IEEE <NUM> standards and the corresponding devices support receiving multiple streams in multiple sessions, the computing device <NUM> can receive both the Audio Stream <NUM> and the Audio Stream <NUM> for a period of time to seamlessly switch from the first IoT device 420A to the second IoT device 420B. In some implementations, during the period of time when the computing device <NUM> receives both the Audio Stream <NUM> and the Audio Stream <NUM>, the computing device <NUM> may optionally send a message to the first IoT device 420A to request that the first IoT device 420A begin to gradually decrease the amplitude or volume of the audio until the first IoT device 420A stops streaming the audio.

At <NUM>, the computing device <NUM> begins receiving both the Audio Stream <NUM> from the first IoT device 420A and the Audio Stream <NUM> from the second IoT device 420B for a period of time. At <NUM>, the computing device <NUM> processes both the Audio Stream <NUM> and the Audio Stream <NUM> and seamlessly mixes the two audio streams with little or no glitches or audio loss. While the computing device <NUM> continues to process both audio streams, at <NUM>, the computing device <NUM> sends a message to the first IoT device 420A with a command to stop recording and streaming the audio. At <NUM>, the second IoT device 420B continues to send the Audio Stream <NUM> and, at <NUM>, the computing device <NUM> begins to receive and process only the Audio Stream <NUM> from the second IoT device 420B.

Since the Bluetooth standards and the corresponding devices do not support receiving multiple streams in multiple sessions, when the computing device <NUM>, the first IoT device 420A, and the second IoT device 420B communicate using Bluetooth protocols, the computing device <NUM> cannot receive both the Audio Stream <NUM> and the Audio Stream <NUM> at the same time. Instead, in some implementations, the computing device <NUM> may send a message to the first IoT device 420A with a command to gradually decrease the amplitude or volume of the audio when the computing device <NUM> is preparing to switch to the second IoT device 420B. When the computing device <NUM> detects the communication from the second IoT device 420B, the computing device <NUM> sends a message to the first IoT device 420A with a command to stop sending the Audio Stream <NUM>, and begins receiving the Audio Stream <NUM> from the second IoT device 420B.

In some implementations, the computing device <NUM> may implement a hysteresis timer that can prevent the computing device <NUM> from switching between IoT devices too frequently. In some implementations, the hysteresis timer can be preconfigured with a hysteresis time period and also can be user configurable. The preconfigured hysteresis time period may be any suitable time period, such as <NUM>, <NUM>, <NUM> milliseconds, and the user can change the configuration to any of the suitable time periods. In one example, when the computing device <NUM> determines, at <NUM>, that the FoV <NUM> of the camera of the computing device <NUM> has changed, the computing device <NUM> starts the hysteresis timer. If the hysteresis time period expires without the computing device <NUM> detecting an additional change in the FoV <NUM> of the camera, the computing device <NUM> proceeds to determine whether to switch between IoT devices. However, if an additional change in the FoV <NUM> is detected before the hysteresis time period expires, the computing device <NUM> restarts the hysteresis time period and does not begin the process for switching between the IoT devices.

<FIG> shows an example flowchart for determining and selecting an IoT device for remotely recording and streaming audio during the A/V recording of the subject. The flowchart <NUM> begins at block <NUM>.

At block <NUM>, the computing device <NUM> determines IoT devices that are displayed within the FoV <NUM> of the camera of the computing device <NUM> and that are capable of remotely recording and streaming audio for recording A/V of a subject. For example, the computing device <NUM> obtains a map of the locations of the IoT devices from a master device of a local network. The computing device <NUM> may determine that the IoT devices each have a microphone and are capable of remote audio recording and audio streaming from information obtained from the master device or from information obtained from each of the IoT devices. The computing device <NUM> may display visual indicators within the FoV <NUM> of the camera for each of the IoT devices that are capable of remote audio recording and audio streaming. The visual indicators also indicate the location of each of the IoT devices within the FoV <NUM> o the camera.

At block <NUM>, the computing device <NUM> selects a first IoT device from the IoT devices that are displayed in the FoV <NUM>. The first IoT device includes a microphone that is capable of remotely recording audio of the subject for the A/V recording and supports streaming of the audio to the computing device <NUM>. For example, the computing device <NUM> may determine, within the FoV <NUM> of the camera of the computing device <NUM>, distance measurements from each of the IoT devices to the focal point <NUM> of the FoV <NUM>. The computing device <NUM> may select a first IoT device from the IoT devices based, at least in part, on the distance measurements. For example, the computing device <NUM> may determine that the first IoT device is located the closest to the focal point <NUM> of the FoV <NUM> compared to the rest of the IoT devices, and therefore may select the first IoT device to remotely record and stream the audio of the subject. In another example, the computing device <NUM> may determine and compare microphone types and performance characteristics associated with each microphone of the IoT devices. The computing device <NUM> may select the first IoT device after determining the first IoT device has a preferred microphone type and performance characteristics, as described in <FIG>. In another example, the computing device <NUM> may select the first IoT device based on both the distance measurements and the microphone type and performance characteristics. Also, in another example, the computing device <NUM> may select the first IoT device based on input received from the user.

At block <NUM>, the computing device <NUM> receives an audio stream from the first IoT device after the computing device <NUM> selects the first IoT device. The received audio stream includes audio of the subject that is remotely recorded by the first IoT device. The computing device <NUM> also records video of the subject using the camera of the computing device <NUM>. The computing device <NUM> generates the A/V recording of the subject using the video recorded by the camera of the computing device <NUM> and the audio remotely recorded by the first IoT device.

<FIG> shows an example flowchart for identifying and selecting an IoT device that is capable of remotely recording and streaming audio during the A/V recording of the subject. The flowchart <NUM> begins at block <NUM>.

At block <NUM>, the computing device <NUM> starts a camera application. For example, the computing device <NUM> starts the camera application in response to receiving user input.

At block <NUM>, the computing device <NUM> obtains a map of the locations of the IoT devices in the IoT environment from a master device of a local network.

At block <NUM>, the computing device <NUM> overlaps (or merges) the map with the FoV <NUM> of the camera of the computing device <NUM>.

At block <NUM>, the computing device <NUM> optionally starts to record audio for the A/V recording using the local microphone of the computing device <NUM>. In some implementations, if the user starts the A/V recording before the computing device <NUM> has time to select one of the remote IoT devices, the camera of the computing device <NUM> initially starts recoding the audio using the local microphone of the computing device <NUM>. After the computing device <NUM> selects one of the IoT devices, the computing device <NUM> seamlessly switches to using the audio that is streamed by the selected IoT device for the A/V recording. In some implementations, the computing device <NUM> may have enough time to select the IoT device before the user starts the A/V recording. Thus, in some cases, the computing device <NUM> may begin the A/V recording using the audio that is streamed from the selected loT device without initially using the local microphone of the computing device <NUM>.

At block <NUM>, the computing device <NUM> determines the number of IoT devices that have a microphone and that are capable of remotely recording and streaming audio. The computing device <NUM> may determine the number of IoT devices that have a microphone and that are capable of remotely recording and streaming audio from information obtained from the master device or from information obtained from each of the IoT devices. In some implementations, the computing device <NUM> displays a visual indication within the FoV <NUM> of the camera of the computing device <NUM> that indicates the location of the IoT devices that have a microphone and that are capable of remotely recording and streaming audio.

At block <NUM>, if the computing device <NUM> determines that the number of IoT devices that have a microphone and that are capable of remotely recording and streaming audio is greater than one, then the flow continues at block <NUM> of <FIG>. Otherwise, if the computing device <NUM> determines that there is only one IoT device that has a microphone and that is capable of remotely recording and streaming audio, then the flow continues at block <NUM>.

At block <NUM>, the computing device <NUM> determines there is one available IoT device, selects and connects with the available IoT device, and begins receiving an audio stream from the IoT device for the A/V recording. If the computing device <NUM> initially started recording audio using the local microphone (at optional block <NUM>), then the computing device <NUM> switches from recording the audio using the local microphone to recording the audio using the microphone of the IoT device.

<FIG> is a continuation of the example flowchart of <FIG> for identifying and selecting IoT devices that are capable of remotely recording and streaming audio during the A/V recording of the subject. The flowchart <NUM> continues at block <NUM>.

At block <NUM>, the computing device <NUM> select one of the available IoT devices having a microphone and that are capable of remotely recording and streaming audio. The computing device <NUM> may select the IoT device based on distance measurements between the IoT devices and the focal point <NUM> of the FoV <NUM>, based on the microphone types and performance characteristics associated with each of the microphones, based on the quality of the audio streams received from the IoT devices, or based on an input received from the user, as described in <FIG>.

At block <NUM>, the computing device <NUM> begins receiving an audio stream from a first IoT device that is selected by the computing device <NUM> for the A/V recording. If the computing device <NUM> initially started recording audio using the local microphone (at optional block <NUM>), then the computing device <NUM> switches from recording the audio using the local microphone to recording the audio using the microphone of the first IoT device.

At block <NUM>, the computing device <NUM> determines whether the FoV <NUM> of the camera of the computing device <NUM> has changed. If the FoV <NUM> of the camera has not changed, the flow returns to block <NUM> and the computing device <NUM> continues receiving the audio stream from the selected IoT device. If the FoV <NUM> of the camera has changed, the flow continues at block <NUM>.

At block <NUM>, the computing device <NUM> determines whether to switch to receiving an audio stream from a second IoT device. For example, if the computing device <NUM> is currently receiving an audio stream from the first IoT device, the computing device <NUM> determines whether to continue receiving the audio stream from the first IoT device or to switch to receiving an audio stream from a second IoT device. The computing device <NUM> may determine to switch to receiving the audio stream from the second IoT device if the second IoT device is closer to the focal point of the FoV <NUM> than the first IoT device after the change in the FoV <NUM> of the camera. If the computing device <NUM> determines not to switch to receiving audio from a second IoT device, the flow returns to block <NUM> and the computing device <NUM> continues receiving the audio stream from the first IoT device. If the computing device <NUM> determines to switch to receiving audio from the second IoT device, the flow continues at block <NUM>.

At block <NUM>, the computing device <NUM> begins to receive an audio stream from the second IoT device for the A/V recording. The computing device <NUM> may implement techniques to coordinate the change from the first IoT device to the second IoT device to seamlessly switch from receiving and processing the audio stream from the first IoT device to receiving and processing the audio stream from the second IoT device for the A/V recording.

In some implementations, the computing device <NUM> may select two or more remote IoT devices to remotely record and stream audio of the subject for the A/V recording. The computing device <NUM> may use the same techniques described for selecting one of the IoT devices to select two or more IoT devices. In one example, within the FoV <NUM> of the camera of the computing device <NUM>, the computing device <NUM> may measure the distances from each of the IoT devices to the focal point <NUM> of the FoV <NUM>, and use the distance measurements to select two or more of the IoT devices. For example, the computing device <NUM> may select the two IoT devices that are closest to the focal point <NUM> based on the distance measurements. In another example, the computing device <NUM> may select two or more of the IoT devices based on the microphone type and performance characteristics associated with each microphone of each of the IoT devices. For example, the computing device <NUM> may compare the microphone types and performance characteristics of all the microphones and may select the two IoT devices that have the preferred microphone types and performance characteristics. In another example, the computing device <NUM> may receive an audio stream from each of the IoT devices, and the computing device <NUM> may process the audio streams to select two or more of the IoT devices. For example, the computing device may select the two IoT devices that streamed the best quality audio. In another example, the computing device <NUM> also may consider the proximity of the IoT devices to each other when selecting two or more IoT devices. For example, if the computing device <NUM> determines there are multiple IoT devices within the focus area <NUM> of the FoV <NUM> of the camera, the computing device <NUM> may select the two IoT devices that are closest to one another. The computing device <NUM> may use distance measurements (similarly as described in <FIG>) to determine the IoT devices that are closest to one another. Thus, in addition to being located in close proximity to the focal point <NUM> of the FoV, the selected IoT devices also may be in close proximity to each other. Furthermore, the computing device <NUM> may implement the same techniques described for switching between microphones when the FoV <NUM> of the camera changes. For example, when the FoV <NUM> of the camera changes, the computing device <NUM> may determine whether the two IoT devices are still the closest two IoT devices to the focal point <NUM> of the FoV <NUM>. If not, the computing device <NUM> may switch one or both of the IoT devices with the corresponding number of IoT devices that are closer to the focal point <NUM> of the FoV <NUM>.

In some implementations, when the devices communication using protocols from the IEEE <NUM> standards, the computing device <NUM> can implement a three-channel remote audio recording and audio streaming. In this implementation, the computing device <NUM> may receive an audio stream from three different microphones of three different IoT devices to improve noise cancellation and suppression. For example, the computing device <NUM> can select a primary microphone and two secondary microphones, with each microphone providing a separate audio stream to the computing device <NUM>. In this example, the computing device <NUM> can receive the three audio streams, apply noise cancellation and suppression techniques, and combine and mix the three audio streams when generating the audio for the A/V recording. In one example, the primary microphone may be used to record the audio and at least one secondary microphone can be used for noise cancellation and suppression.

In some implementations, instead of the computing device <NUM> receiving the audio stream from one or more remote IoT devices, the master device of the local network may receive the one or more audio streams from the one or more remote IoT devices, process the audio streams, and provide the processed audio to the computing device <NUM>. In one example, the computing device <NUM> sends a message to one or more remote IoT devices having microphones with a command to begin recording audio and to stream the audio to the master device. In another example, the computing device <NUM> provides the computing device's location and the FoV <NUM> of the computing device's camera to the master device, and the master device determines which IoT devices to enable to begin recording and streaming audio based on the computing device's location and the FoV <NUM> of the camera.

<FIG> shows an example flowchart for establishing a communication link between a first computing device of a first user and a second computing device of a second user after detecting the second computing device within a FoV of a camera of the first computing device. The flowchart <NUM> begins at block <NUM>.

At block <NUM>, the first computing device of a first user detects a second computing device of a second user that is displayed within a FoV of a camera of the first computing device. In one example, if both the first computing device and the second computing device are pairs of smart glasses (or AR glasses), when the first user looks in the direction of the second user from a distance, the first user's smart glasses may detect the second user's smart glasses in the FoV of the camera of the first user's smart glasses. When the first user looks in the direction of the second user, the first user may position the second user wearing the smart glasses in the FoV of the camera of the first user's smart glasses, and then the first user may provide an input indicating the first user wants to communicate with the second user.

At block <NUM>, the first computing device provides a request message to the second computing device to request approval to establish a communication link between the first computing device and the second computing device, in response to detection of the second computing device within the FoV of the camera of the first computing device. For example, the first user's smart glasses can send a request message to the second user's smart glasses to request approval to establish a communication link between the two devices. For example, the first user's smart glasses can send the request message using protocols in the IEEE <NUM> or Bluetooth standards.

At block <NUM>, the first computing device receives a response message from the second computing device granting the request to establish the communication link. For example, the first user's smart glasses may receive a response message from the second user's smart glasses accepting the request to establish the communication link after the second user's smart glasses detects input from the second user indicating acceptance of the request. In another example, the first user's smart glasses may receive a response message denying the request to establish the communication link.

At block <NUM>, the first computing device establishes the communication link between the first computing device and the second computing device for the first user to communication with the second user. For example, the first user's smart glasses and the second user's smart glasses exchange messages to establish a two-way communication link. After the communication link is established, the first user can communication with the second user by speaking into the microphone of the first user's smart glasses and the second user will hear the first user via the speakers of the second user's smart glasses, and vice versa.

In some implementations, after the communication link between the first computing device and the second computing device is established, either the first user or the second user can try to establish a communication link with a third computing device of a third user in order to start a conference call between the three users. For example, the first user can look in the direction of the third user and position the third user wearing the smart glasses in the FoV of the camera of the first user's smart glasses. Then, the first user may provide an input indicating the first user wants to communicate with the second user and the first computing device may send a request message to the third user. Similar to the process described with respect to the second user, the third user can grant or deny the request to establish a communication link.

In some implementations, if the first user is wearing smart glasses but the second user does not have smart glasses, the first user can place the second user's mobile phone or another IoT device close to the second user in the FoV of the camera of the smart glasses to try to establish a communication link. If the IoT device only has a microphone, then the first user's smart glasses can connect with the microphone of the IoT device. For example, a speaker in an auditorium wearing smart glasses can connect to an IoT device having a microphone that is close to an audience member that is asking a question, so that the speaker can hear the audience member's question via the microphone. In another example, a first user wearing smart glasses can communicate from a distance with a second user having a mobile phone if the second user accepts the communication request via the mobile phone.

In some implementations, the first user wearing smart glasses can place a product at a location (such as a showroom, trade show, or store) in the FoV of the camera of the smart glasses to dynamically obtain product information from a distance. For example, when the first user looks in the direction of the product and places the product in the FoV of the camera of the smart glasses, the smart glasses can establish a communication link with the product and dynamically obtain product information (either directly from the product or from a master device). For example, after establishing the communication link with the product, the user can send voice commands to the product via the communication link requesting the product information. Also, the first user can obtain product information from many products from a distance by establishing communication links with the different products from a distance and requesting the information using voice commands.

<FIG> shows a block diagram of an example electronic device <NUM> for implementing aspects of this disclosure. In some implementations, the electronic device <NUM> may be similar to the computing device <NUM>. The electronic device <NUM> may be a laptop computer, a tablet computer, a mobile phone, a gaming console, a smartwatch, virtual or augmented reality device (such as a pair of smart glasses), a drone, or another electronic system. The electronic device <NUM> includes a processor <NUM> (possibly including multiple processors, multiple cores, multiple nodes, or implementing multi-threading, etc.). The electronic device <NUM> includes a memory <NUM>. The memory <NUM> may be system memory or any one or more of the below-described possible realizations of a machine-readable medium or computer-readable medium. The electronic device <NUM> also may include a bus <NUM> (such as PCI, ISA, PCI-Express, HyperTransport®, InfiniBand®, NuBus, AHB, AXI, etc.). The electronic device may include one or more network interfaces <NUM>, which may be a wireless network interface (such as a WLAN interface, a Bluetooth® interface, a WiMAX interface, a ZigBee® interface, a Wireless USB interface, etc.) or a wired network interface (such as a powerline communication interface, an Ethernet interface, etc.). In some implementations, electronic device <NUM> may support multiple network interfaces <NUM>, each of which may be configured to couple the electronic device <NUM> to a different communication network.

The memory <NUM> includes functionality to support various implementations. The memory <NUM> may include one or more functionalities that facilitate implementation of identifying and selecting one or more IoT devices for remotely recording and streaming audio during an A/V recording. For example, memory <NUM> can implement one or more aspects of the computing device <NUM>. The memory <NUM> can embody functionality to enable implementations described in <FIG>. The electronic device <NUM> also may include additional components, such as a camera module <NUM>, a microphone module <NUM>, a user interface <NUM>, and other input/output component.

Any one of these functionalities may be partially (or entirely) implemented in hardware, such as on the processor <NUM>. For example, the functionality may be implemented with an application specific integrated circuit, in logic implemented in the processor <NUM>, in a co-processor on a peripheral device or card, etc. Further, realizations may include fewer or additional components not illustrated in <FIG> (such as video cards, audio cards, additional network interfaces, peripheral devices, etc.). The processor <NUM> and the memory <NUM> may be coupled to the bus <NUM>. Although illustrated as being coupled to the bus <NUM>, the memory <NUM> may be directly coupled to the processor <NUM>.

The interchangeability of hardware and software has been described generally, in terms of functionality, and illustrated in the various illustrative components, blocks, modules, circuits and processes described throughout. Whether such functionality is implemented in hardware or software depends on the particular application and design constraints imposed on the overall system.

The hardware and data processing apparatus used to implement the various illustrative logics, logical blocks, modules and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or, any conventional processor, controller, microcontroller, or state machine. A processor also may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry that is specific to a given function.

By way of example, and not limitation, such computer-readable media may include cache memory, RAM (including SRAM, DRAM, zero capacitor RAM, Twin Transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM, or the like), ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray™ disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations also can be included within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and instructions on a machine-readable medium and computer-readable medium, which may be incorporated into a computer program product.

Certain features that are described in this specification in the context of separate implementations also can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation also can be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Claim 1:
A method (<NUM>) for network communication performed by a computing device for recording audio and video, A/V, comprising:
determining (<NUM>), by the computing device, a plurality of internet of things, IoT, devices that are displayed within a field of view, FoV, of a camera of the computing device and that are capable of remote audio recording and audio streaming;
selecting (<NUM>), by the computing device, a first IoT device from the plurality of IoT devices, the first IoT device having a first microphone that is capable of remotely recording audio of a subject for the computing device;
receiving (<NUM>), by the computing device, an audio stream from the first IoT device after selecting the first IoT device, the audio stream including the audio of the subject remotely recorded by the first IoT device for the computing device; and
wherein selecting the first IoT device from the plurality of IoT devices comprises:
determining a microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices;determining, within the FoV of the camera of the computing device, distance measurements from each of the plurality of IoT devices to a focal point of the FoV;
comparing the microphone type and performance characteristics associated with each microphone of each of the plurality of IoT devices; and
selecting the first IoT device from the plurality of IoT devices based on the microphone type and performance characteristics associated with the plurality of IoT devices so that an IoT device with a preferred microphone type and performance characteristics is selected;
wherein, if two or more IoT devices from the plurality of IoT devices have the same preferred microphone type and performance characteristics, the selecting further comprises:
selecting the first IoT device from the plurality of IoT devices based on the distance measurements associated with the two or more IoT devices from the plurality of IoT devices so that an IoT device closest to the focal point of the FoV is selected.