System and method for providing enhanced audio in a video environment

A method is provided in one example and includes receiving audio data at a microphone array that includes a plurality of microphones. The microphone array is provisioned at a first endpoint, which includes a camera element configured to capture video data associated with a video session involving the first endpoint and a second endpoint. The method also includes formatting the audio data into a time division multiplex (TDM) stream, and communicating the stream to a port for a subsequent communication over a network and to the second endpoint.

TECHNICAL FIELD

This disclosure relates in general to the field of communications and, more particularly, to providing enhanced audio in a video environment.

BACKGROUND

Video services have become increasingly important in today's society. In certain architectures, service providers may seek to offer sophisticated video conferencing services for their end users. The video conferencing architecture can offer an “in-person” meeting experience over a network. Video conferencing architectures can deliver real-time, face-to-face interactions between people using advanced visual, audio, and collaboration technologies. The ability to optimize video communications provides a significant challenge to system designers, device manufacturers, and service providers alike.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

A method is provided in one example and includes receiving audio data at a microphone array that includes a plurality of microphones. The microphone array is provisioned at a first endpoint, which includes a camera element configured to capture video data associated with a video session involving the first endpoint and a second endpoint. The method also includes formatting the audio data into a time division multiplex (TDM) stream, and communicating the stream to a port for a subsequent communication over a network and to the second endpoint.

In more specific implementations, the plurality of microphones is logarithmically spaced from each other, and the microphone array includes Micro Electrical-Mechanical System (MEMS) microphones. A plurality of audio codecs can be configured to convert a digital output from the microphone array into a different format for interfacing with a transmitter integrated circuit.

In detailed embodiments, the method can include measuring a delay associated with the audio data in order to cancel echoing effects associated with the video session. The delay is measured by playing music during an initial startup phase associated with a video conferencing platform. The delay can be based on a type of display being used to render images associated with the video session. An embedded watermark signature provided in speakers of the microphone array can be used to determine a delay for the audio data. The delay for the audio data can be dynamically updated based on a clocking mechanism associated with the display.

Example Embodiments

Turning toFIG. 1,FIG. 1is a simplified block diagram of a system10for providing a video session in a network environment. In this particular example, system10may include a display12, a camera element14, a user interface (UI)18, a console element20, a handset28, and a network30. A series of speakers16are provisioned in conjunction with camera element14in order to transmit and receive audio data. In one particular example implementation, a wireless microphone24is provided in order to receive audio data in a surrounding environment (e.g., from one or more audience members). Note that this wireless microphone24is purely optional, as speakers16are capable of sufficiently capturing audio data in a surrounding environment during any number of videoconferencing applications (which are detailed below).

In general terms, system10can be configured to capture video image data and/or audio data in the context of videoconferencing. System10may include a configuration capable of transmission control protocol/internet protocol (TCP/IP) communications for the transmission and/or reception of packets in a network. System10may also operate in conjunction with a user datagram protocol/IP (UDP/IP) or any other suitable protocol, where appropriate and based on particular communication needs.

The term ‘handset’ is broad and, accordingly, it can encompass any number of devices used to manage, direct, or otherwise control certain aspects of system10. In certain implementations, handset28can be used as a remote control for system10. For example, handset28can offer a wireless remote control that allows it to communicate with display12, camera element14, and/or console element20via a wireless network link (e.g., infrared, Bluetooth, any type of IEEE 802.11-based protocol, etc.). Handset28can further be provisioned as a wireless mobile phone (e.g., a speakerphone device) with various dial pads: some of which are shown by way of example inFIG. 1. In other implementations, handset28operates as a learning mechanism and/or a universal remote controller, which allows it to readily control display12, camera element14, console element20, and/or any audiovisual (AV) receiver device (e.g., managing functions such as ON/OFF, volume, input select, etc. to enhance the overall video experience). In a particular set of examples, a specific button on handset28can launch UI18for navigating through any number of options provided in submenus of the UI software. Additionally, a dedicated button can be used to make/answer calls, end calls, turn on/off camera element14, turn on/off the microphone on, turn on/off console element20, etc. Furthermore, a set of playback controls can be provided on handset28in order to control the video data being rendered on display12.

Note that handset28can be configured to launch, control, and/or manage UI18. In one particular instance, UI18includes a clover design having four separate functions along its perimeter (i.e., up, down, left, right). The center of UI18can be used to initiate calls or to configure call options. The lower widget icon may be used to adjust settings, inclusive of controlling profile information, privacy settings, console settings, etc. The right-hand icon (when selected) can be used to view video messages sent to a particular user. The upper icon can be used to manage contacts (e.g., add, view, and connect to other individuals). The director's card (provided as the left icon) can be used to record and send video messages to other individuals. It is imperative to note that these menu choices can be changed considerably without departing from the scope of the present disclosure. Additionally, these icons may be customized, changed, or managed in any suitable fashion. Furthermore, the icons of UI18are not exhaustive, as any other suitable features may be provided in the context of UI18. Along similar lines, the submenu navigation choices provided beneath each of these icons can include any suitable parameter applicable to videoconferencing, networking, user data management, profiles, etc.

In operation of an example implementation, system10can be used to conduct video calls (e.g., supporting both inbound and outbound directional call flows). For the inbound call scenario, on reception of an inbound call request, console element20is configured to contact the paired handset(s)28(e.g., waking it from sleep, where appropriate). Handset28can be configured to play a ringtone, turn on an LED indicator, and/or display UI18(e.g., including the incoming caller's contact information). If configured to do so, UI18can also be displayed over any passthrough video sources on console element20. If the callee chooses to answer the call with one of the call control buttons, console element20offers its media capabilities to the caller's endpoint. In certain example implementations, by default, audio media can be offered at the start of the call. At any time during a voice call, both parties can agree to enter into a full video session (e.g., referred to as a “go big” protocol) at which point video media is negotiated. As a shortcut, the intention to “go big” can be pre-voted at the start of the call. At any time after video media is flowing, the call can also be de-escalated back to an audio-only call. In certain instances, there could be an option to automatically answer incoming calls as immediate full-video sessions.

In the case of an ad hoc outbound call, the user can select a callee from their contact list, select a callee via a speed dial setting, or alternatively the user can enter any type of identifier (e.g., a telephone number, a name, a videoconferencing (e.g., Telepresence, manufactured by Cisco, Inc. of San Jose, Calif.) number directly). If the callee answers, the call scenario proceeds, similar to that of an inbound call. In the case of a hold and resume scenario, an in-call UI18signal can be provided to put a call on hold, and subsequently the call can be resumed at a later time. Note that in other instances, system10can be used to execute scheduled calls, call transfer functions, multipoint calls, and/or various other conferencing capabilities.

In the case of the consumer user attempting a communication with a business entity, certain parameters may be changed based on interoperability issues. For example, secure business endpoints may be supported, where signaling and media would be secure (both audio and video). Appropriate messages can be displayed in UI18to inform the user of the reason for any security-forced call drops. Signaling can be considered secure by having both a business exchange and consumer networks physically co-located, or by using a secure tunnel (e.g., a site-to-site virtual private network (VPN) tunnel) between the two entities.

Before turning to additional flows associated with system10,FIG. 2is introduced in order to illustrate some of the potential arrangements and configurations for system10. In the particular example implementation ofFIG. 2, camera element14includes a processor40aand a memory element42a. Camera element14is coupled to console element20, which similarly includes a processor40band a memory element42b. A power cord36is provided between an outlet and console element20. Any suitable connections (wired or wireless) can be used in order to connect any of the components ofFIG. 2. In certain examples, the cables used may include Ethernet cables, High-Definition Multimedia Interface (HDMI) cables, universal serial bus (USB) cables, or any other suitable link configured for carrying data or energy between two devices.

In regards to a physical infrastructure, camera element14can be configured to fasten to any edge (e.g., a top edge) of display12(e.g., a flat-screen HD television). Camera element14can be included as part of an integrated component (i.e., a single component, a proprietary element, a set-top box, console element20, etc.) that could include speakers16(e.g., an array microphone). Thus, all of these elements (camera element14, speakers16, console element20) can be combined and/or be suitably consolidated into an integrated component that rests on (or that is fixed to, or that is positioned near) display12. Alternatively, each of these elements are their own separate devices that can be coupled (or simply interact with each other), or be adequately positioned in any appropriate fashion.

Also provided inFIG. 2are a router34and a set-top box32: both of which may be coupled to console element20. In a particular example, router34can be a home wireless router configured for providing a connection to network30. Alternatively, router34can employ a simple Ethernet cable in order to provide network connectivity for data transmissions associated with system10. Handset28can be recharged through a cradle dock26(as depicted inFIG. 2). [Handset28can be functional while docked.] Alternatively, handset28may be powered by batteries, solar charging, a cable, or by any power source, or any suitable combination of these mechanisms.

In one particular example, the call signaling of system10can be provided by a session initiation protocol (SIP). In addition, the media for the videoconferencing platform can be provided by Secure Real-time Transport Protocol (SRTP), or any other appropriate real-time protocol. SRTP addresses security for RTP and, further, can be configured to add confidentiality, message authentication, and replay protection to that protocol. SRTP is preferred for protecting voice over IP (VoIP) traffic because it can be used in conjunction with header compression and, further, it generally has no effect on IP quality of service (QoS). For network address translation (NAT)/firewall (FW) traversal, any suitable mechanism can be employed by system10. In one particular example, these functions can be provided by a split-tunneled VPN with session traversal utilities for NAT (STUN) and Interactive Connectivity Establishment (ICE).

Signaling can propagate to a call agent via the VPN. Additionally, media can be sent directly from the endpoint to another endpoint (i.e., from one videoconferencing platform to another). Note that as used herein, the term ‘media’ is inclusive of audio data (which may include voice data) and video data (which may include any type of image data). The video data can include any suitable images (such as that which is captured by camera element14, by a counterparty's camera element, by a Webcam, by a smartphone, by an iPad, etc.). The term ‘smartphone’ as used herein includes any type of mobile device capable of operating in conjunction with a video service. This would naturally include items such as the Google Droid, the iPhone, an iPad, etc. In addition, the term ‘signaling data’ is inclusive of any appropriate control information that can be sent toward a network. This may be inclusive of traffic used to establish a video session initially, along with any type of negotiations (e.g., for bit rates, for bandwidth, etc.) that may be appropriate for the particular videoconference. This may further be inclusive of items such as administrative traffic, account traffic (for user account management, contact lists [which include buddy lists, as detailed below], etc.), and/or other types of traffic, which are not provided as part of the media data.

In order to handle symmetric NAT, Traversal Using Relay NAT (TURN) can be used by system10in particular embodiments. User names for the videoconferencing platform can be provided by E.164 numbers in a particular example. Alternatively, the user naming can be a simple user ID (e.g., assigned by the service provider, selected by the user, etc.), a full name of the user (or a group name), an avatar, or any other symbol, number, or letter combination that can be used to distinguish one user from another. Note that a single name can also be associated with a group (e.g., a family, a business unit, etc.). The security for communications of system10can be addressed a number of ways. In one implementation, the video services (i.e., cloud services) can be protected by any suitable security protocol (e.g., security software, adaptive security appliances (ASA), etc.). Additionally, intrusion protection systems, firewalls, anti-denial of service mechanisms can be provided for the architecture (both out in the network, and/or locally within a residential environment).

Turning to details associated with the infrastructure of system10, in one particular example, camera element14is a video camera configured to capture, record, maintain, cache, receive, and/or transmit image data. This could include transmitting packets over network30to a suitable next destination. The captured/recorded image data could be stored in camera element14itself, or be provided in some suitable storage area (e.g., a database, a server, console element20, etc.). In one particular instance, camera element14can be its own separate network device and have a separate IP address. Camera element14could include a wireless camera, a high-definition camera, or any other suitable camera device configured to capture image data.

Camera element14may interact with (or be inclusive of) devices used to initiate a communication for a video session, such as a switch, console element20, a proprietary endpoint, a microphone, a dial pad, a bridge, a telephone, a computer, or any other device, component, element, or object capable of initiating video, voice, audio, media, or data exchanges within system10. Camera element14can also be configured to include a receiving module, a transmitting module, a processor, a memory, a network interface, a call initiation and acceptance facility such as a dial pad, one or more displays, etc. Any one or more of these items may be consolidated, combined, eliminated entirely, or varied considerably and those modifications may be made based on particular communication needs.

Camera element14can include a high-performance lens and an optical zoom, where camera element14is capable of performing panning and tilting operations. The video and the audio streams can be sent from camera element14to console element20, where they are mixed into the HDMI stream. In certain implementations, camera element14can be provisioned as a light sensor such that the architecture can detect whether the shutter of the camera is open or closed (or whether the shutter is partially open.) An application program interface (API) can be used to control the operations of camera element14.

Display12offers a screen on which video data can be rendered for the end user. Note that as used herein in this Specification, the term ‘display’ is meant to connote any element that is capable of delivering image data (inclusive of video information), text, sound, audiovisual data, etc. to an end user. This would necessarily be inclusive of any panel, plasma element, television (which may be high-definition), monitor, computer interface, screen, Telepresence devices (inclusive of Telepresence boards, panels, screens, surfaces, etc.), or any other suitable element that is capable of delivering/rendering/projecting such information.

Network30represents a series of points or nodes of interconnected communication paths for receiving and transmitting packets of information that propagate through system10. Network30offers a communicative interface between any of the components ofFIGS. 1-2and remote sites, and may be any local area network (LAN), wireless local area network (WLAN), metropolitan area network (MAN), wide area network (WAN), VPN, Intranet, Extranet, or any other appropriate architecture or system that facilitates communications in a network environment.

Console element20is configured to receive information from camera element14(e.g., via some connection that may attach to an integrated device (e.g., a set-top box, a proprietary box, etc.) that sits atop (or near) display12and that includes (or is part of) camera element14). Console element20may also be configured to control compression activities, or additional processing associated with data received from camera element14. Alternatively, the actual integrated device can perform this additional processing before image data is sent to its next intended destination. Console element20can also be configured to store, aggregate, process, export, or otherwise maintain image data and logs in any appropriate format, where these activities can involve processor40band memory element42b. Console element20is a video element that facilitates data flows between endpoints and a given network. As used herein in this Specification, the term ‘video element’ is meant to encompass servers, proprietary boxes, network appliances, set-top boxes, or other suitable device, component, element, or object operable to exchange video information with camera element14.

Console element20may interface with camera element14through a wireless connection, or via one or more cables or wires that allow for the propagation of signals between these elements. These devices can also receive signals from an intermediary device, a remote control, handset28, etc. and the signals may leverage infrared, Bluetooth, WiFi, electromagnetic waves generally, or any other suitable transmission protocol for communicating data (e.g., potentially over a network) from one element to another. Virtually any control path can be leveraged in order to deliver information between console element20and camera element14. Transmissions between these two devices can be bidirectional in certain embodiments such that the devices can interact with each other. This would allow the devices to acknowledge transmissions from each other and offer feedback where appropriate. Any of these devices can be consolidated with each other, or operate independently based on particular configuration needs. In one particular instance, camera element14is intelligently powered using a USB cable. In a more specific example, video data is transmitted over an HDMI link, and control data is communicated over a USB link.

In certain examples, console element20can have an independent light sensor provisioned within it to measure the lighting in a given room. Subsequently, the architecture can adjust camera exposure, shuttering, lens adjustments, etc. based on the light that is detected in the room. Camera element14is also attempting to provide this function; however, having a separate light sensor offers a more deterministic way of adjusting these parameters based on the light that is sensed in the room. An algorithm (e.g., within camera element14and/or console element20) can be executed to make camera adjustments based on light detection. In an IDLE mode, the lens of camera element14can close automatically. The lens of camera element14can open for an incoming call, and can close when the call is completed (or these operations may be controlled by handset28). The architecture can also account for challenging lighting environments for camera element14. For example, in the case of bright sunlight behind an individual, system10can optimize the exposure of the individual's face.

The video messaging can include snapshots of video frames that would be indicative of the actual message images. In the user's video Inbox, the current videomail can include images of the actual messages being stored for future playback. For example, if the message were from the user's mother, the videomail would include a series of snapshots of the mother speaking during that videomail. In one particular example, the actual videomail is sampled at certain time intervals (e.g., every 10 seconds) in order to generate these images, which serve as a preview of the videomail message. Alternatively, the snapshots can be limited in number. In other instances, the snapshots are arbitrarily chosen, or selected at the beginning, the middle, and the end of the video message. In other implementations, the snapshots are taken as a percentage of the entire video message (e.g., at the 20% mark, at the 40% mark, and at the 100% mark). In other examples, the videomail in the Inbox is previewed by just showing the image associated with that particular user ID that authored the video message.

In operation of an example involving a user watching a normal television program on display12, an incoming call can be received by the videoconferencing platform. The notification can arrive even if the television is off (e.g., through speakers of system10). If an individual chooses to answer the call, then the videoconferencing platform takes over the television. In one example involving a digital video recorder (DVR), the programming can be paused. In other examples, the user can keep the call minimized so (for example) a user could speak with a friend while watching a football game. Console element20can be configured to record a message, and then send that message to any suitable next destination. For example, the user can send a link to someone for a particular message. The user can also use Flip Share or YouTube technology to upload/send a message to any appropriate destination. In a general sense, the messages can be resident in a network cloud such that they could still be accessed (e.g., over a wireless link) even if the power were down at the residence, or if the user were not at the residence.

The user can also switch from a video call to handset28, and from handset28back to a video call. For example, the user can initiate a call on a smartphone and subsequently transition it to the videoconferencing display12. The user can also do the reverse, where the user starts at the videoconferencing platform and switches to a smartphone. Note that wireless microphone24can operate in a certain, preferred range (e.g., 12 to 15 feet), where if the individual moves further away from that range, users could elect to transition to handset28(in a more conventional telephony manner). Consider the case where the room becomes noisy due to family members, and the user on the videoconferencing call elects to simply switch over to a smartphone, to a given landline, etc.

Motion detection can also be used in order to initiate, or to answer video calls. For example, in the case where a remote control is difficult to find in a living room, a simple hand-waving gesture could be used to answer an incoming video call. Additionally, the system (e.g., camera element14cooperating with console element20) can generally detect particular body parts in order to execute this protocol. For example, the architecture can distinguish between a dog running past display12, versus handwaving being used to answer an incoming call. Along similar lines, the user can use different gestures to perform different call functions (e.g., clasping his hands to put a call on hold, clapping his hands to end the call, pointing in order to add a person to a contact list, etc.).

Note that Wi-Fi is fully supported by system10. In most videoconferencing scenarios, there can be massive amounts of data (much of which is time critical) propagating into (or out of) the architecture. Video packets (i.e., low-latency data) propagating over a Wi-Fi connection can be properly accommodated by system10. In one particular example, nonmoving (static) background images can be segmented out of the video image, which is being rendered by display12. The architecture (e.g., through console element20) can then lower the bit rate significantly on those images. Allocations can then be made for other images that are moving (i.e., changing in some way). In certain example implementations, face-detection algorithms can also be employed, where the video is optimized based on those algorithm results.

Certain phone features allow for handset28to offer speed dialing, and a mechanism for saving contacts into a contact list. Calls can be made to users on the speed dial list or the contact list with a single button push on handset28. Additionally, calls can be initiated using either the UI of handset28, or the on-screen UI18. Furthermore, calls can be initiated from a web portal, where the caller can confirm call initiation at the endpoint by pressing voice-only, or a video call button on handset28. Also, calls can be initiated from other web pages via a call widget (e.g., calling a person by clicking on his Facebook object). In addition, the caller can look up a recipient in an online directory (e.g., a directory of all Telepresence users stored in a database), place a call to that recipient, and save the recipient's contact information into the contact list. In terms of receiving videoconferencing calls, incoming calls can be accepted with a single button push on handset28. Call recipients have the opportunity to accept or reject a call. Rejected calls can be routed to videomail (if permitted by the recipient's safety settings).

In regards to call quality, if the available bandwidth decreases during a call, the video resolution is scaled down, as appropriate. If the available bandwidth increases during a call, the video resolution can be scaled up. An on-screen icon can be provided on display12to inform the user of the quality of his videoconferencing experience. The purpose of this information can be to inform the user of a poor experience, potentially being caused by network conditions, and that the user can improve his experience by upgrading his broadband service. When communicating with a Webcam, the picture on display12can be windowed inside a black frame: regardless of the actual quality of the Webcam video.

In regards to videomail, when a call cannot be answered in real time, it is not lost, but rather, forwarded automatically to videomail. Videomail can be accessed from the videoconferencing system, a web portal, a smartphone, laptop, or any other suitable endpoint device to be used by a user. Note that the user is afforded the ability to set a designated interval for when an incoming counterparty would be relegated to the user's videomail Inbox. The term ‘designated interval’ is inclusive of a number of rings, a certain time period (e.g., in seconds), or a zero interval, in which case the counterparty's video call request would be immediately routed to the user's videomail. In certain embodiments, the ‘designated interval’ has a default configured by an administrator.

Videomail can be stored in the network (e.g., in the cloud) in particular implementations of system10. Alternatively, the videomail can be stored locally at the consumer's residence (e.g., at a laptop, a personal computer, an external hard drive, a server, or in any other appropriate data storage device). Videomail can be played with the following minimum set of playback controls: Play, Pause, Stop, Fast or Skip Forward, Fast or Skip Reverse, Go Back to Start. In a particular implementation, videomail is only viewed by the intended recipient. Notifications of new videomail can be sent to other devices by short message service (SMS) text message (e.g., to a mobile device) or by email. An immediate notification can also be shown on handset28. For video recordings, videos can be recorded and stored in the network for future viewing and distribution (e.g., as part of video services, which are detailed below with referenceFIG. 3). Calls can similarly be recorded in real time and stored in the network for future viewing and distribution. When sharing recorded videos with videoconferencing users, the architecture can specify exactly which videoconferencing users have access to the video data. When the share list contains one or more email addresses, access control is not enabled in particular implementations (e.g., any individual who has the URL could access the video).

In terms of media sharing, system10can provide a simple mechanism for sharing digital photos and videos with removable flash media, flash and hard-drive high definition digital camcorders, digital still cameras, and other portable storage devices. This can be fostered by supporting an external USB connection for these devices to the USB port, which can be provisioned at console element20, display12, camera element14, a proprietary device, or at any other suitable location.

The media sharing application (e.g., resident in console element20) supports playback of compressed AV file media that is stored on the USB device. Furthermore, this media sharing can be supported via an external HDMI connection for these devices to the HDMI port. System10can also provide a mechanism for sharing digital photos and videos that are on a computer, on a Network Attached Storage (NAS) device, on the local network, etc. The mechanism can be universal plug and play (UPnP)/digital living network alliance (DLNA) renderer compliant. The media sharing application can also provide a mechanism for sharing digital photos and videos that are on either a photo or video sharing site (e.g., Flickr, YouTube, etc.), as discussed herein.

System10can also provide a mechanism for viewing broadcast HDTV programs (e.g., watching the Superbowl) with the HDTV set-top box HDMI AV feed displayed in picture-in-picture (PIP) with the call video. Continuing with this example, the Super Bowl broadcast feed can be from a local set-top box32and not be shared. Only the call video and voice would be shared in this example. The audio portion of the call can be redirected to handset28(e.g., speakerphone by default). The audio from the local TV can be passed through to HDMI and optical links (e.g., TOSlink outputs).

In an example scenario, initially the game video can fill the main screen and the call video could be in the smaller PIP. The audio for the game can pass through the box to the television, or to AV receiver surround-sound system. The audio for the video call would be supported by handset28. In a different scenario, while watching the game, where one caller prefers to switch the main screen from the game to the video call (e.g., during halftime), then the following activities would occur. [Note that this is consistent with the other PIP experiences.] The call video can fill the main screen, where the game fills the smaller PIP window. The audio for the video call can move to the TV or to the AV receiver surround-sound system, and the game audio can switch to handset28. Note that none of these activities requires the user to be “off camera” to control the experience: meaning, the user would not have to leave his couch in order to control/coordinate all of these activities.

In one particular example, console element20and camera element14can support any suitable frame rate (e.g., a 50-60 frames/second (fps) rate) for HD video for local, uncompressed inputs and outputs. Additionally, the video (e.g., the HDMI 1.3 video) can be provided as a digital signal input/output for local, uncompressed inputs and outputs. There is a passthrough for high-bandwidth Digital Content Protection (HDCP) data for local, uncompressed inputs and outputs from HDMI.

In regards to audio support, HDMI audio can be provided as a digital signal input/output. There can also be a stereo analog line-level output to support legacy devices in the environment. This is in addition to a digital audio output, which may be in the form of an optical link output such as a TOSlink output. For the audiovisual switching activities, audio and video can be patched from inputs, videoconferencing video, or other generated sources, to a local full-screen output. The architecture can offer a protocol for automatically turning on and selecting the correct source of the HDTV (along with any external audio system, when the audiovisual configuration allows for this while answering a call). This feature (and the other features of handset28) can be implemented via infrared, Bluetooth, any form of the IEEE 802.11 protocol, HDMI-Consumer Electronics Control (CEC), etc.

In regards to camera element14, the architecture can provide a full-motion video (e.g., at 30 fps). Participants outside of the range may be brought into focus via autofocus. Camera element14can provide identification information to console element20, a set-top satellite, and/or any other suitable device regarding its capabilities. Camera element14can be provisioned with any suitable pixel resolution (e.g., 1280×720 pixel (720p) resolution, 1920×1080 pixel (1080p) resolution, etc.). If depth of focus is greater than or equal to two meters, then manual focus can be suggested for setup activities, and the autofocus feature/option would be desirable for the user. In operation, the user can manually focus camera element14on his sofa (or to any other target area) during setup. If successful, this issue would not have to be revisited. If depth of focus is less than or equal to one meter (which is commonly the case) then autofocus can be implemented. A digital people-action finder may also be provisioned for system10using camera element14. Both pan and tilt features are available manually at setup, and during a video call. Similarly, zoom is available manually at set-up time, and during a video call.

Handset28may be equipped with any suitable microphone. In one particular implementation, the microphone is a mono-channel mouthpiece microphone optimized for capturing high quality audio in a voice range. The microphone may be placed to optimize audio capture with standard ear-mouth distance. Handset28can have a 3.5 mm jack for a headphone with microphone. Note that system10can support Home Network Administration Protocol (HNAP) and, further, be compatible with Network Magic, Linksys Easy-Link Advisor, or any other suitable home network management tool.

In one example, handset28has an infrared transmitter for controlling standard home theatre components. The minimum controls for handset28in this example can be power-on, input select, volume up/down, and audio output mute of the TV and AV receiver. Console element20(along with camera element14) can have an infrared receiver to facilitate pairing of the videoconferencing system with other remote controls, which can allow other remotes to control the videoconferencing system. Suitable pairing can occur either by entering infrared codes into handset28, or by pointing a remote from the target system at an infrared receiver of the videoconferencing system (e.g., similar to how universal remotes learn and are paired).

For call management, system10can allow a user to initiate, accept, and disconnect calls to and from voice-only telephones (e.g., using handset28in a voice-only mode). Call forwarding can also be provided such that video calls are forwarded between console elements20at each endpoint of the video session. Additionally, announcements can be provided such that a default announcement video can be played to callers who are leaving a videomail. A self-view is available at any time, and the self-view can be triggered through a user demand by the user pressing a button on handset28. The self-view can be supported with a mirror mode that shows the reverse image of the camera, as if the user was looking in a mirror. This can occur at any time, including while idle, while on a videoconferencing call, while on an audio-only call, etc.

FIG. 3is a simplified block diagram illustrating one potential operation associated with system10. In this particular implementation, console element20is provisioned with a VPN client module44, and a media module46. Console element20is coupled to a home router48, which can provide connectivity to another videoconferencing endpoint50via a network52. Home router48can also provide connectivity to a network that includes a number of video services56. In this example, video services56include a consumer database58, a videomail server60a call control server62, a web services64, and a session border controller66.

Any number of traffic management features can be supported by system10. In a simple example, system10can allow a point-to-point connection to be made between two home video conferencing systems. A connection can also be made between a home video conferencing system and an enterprise videoconferencing system. The packets associated with the call may be routed through a home router, which can direct the packets to an exchange or a gateway in the network. The consumer endpoint does not need to support the second data channel; any shared content can be merged into the main data stream. A multipoint connection can be made between a combination of three or more home and enterprise videoconferencing systems.

In operation, the VPN is leveraged in order to transmit administrative and signaling traffic to the network. Additionally, the media data (e.g., voice and video) can be exchanged outside of that link (e.g., it can be provisioned to flow over a high bandwidth point-to-point link). This linking can be configured to protect administrative and signaling traffic (which may be inclusive of downloads), while simultaneously conducting high-speed data communications over the point-to-point pathway.

In the particular example ofFIG. 3, secure signaling and administrative data is depicted as propagating between home router48and video services56. A number of VPN ports are also illustrated inFIG. 3. The ports can be associated with any appropriate security protocol (e.g., associated with IPsec, secure socket layer (SSL), etc.). Additionally, media data can propagate between network52and home router48, where RTP ports are being provisioned for this particular exchange involving a counterparty endpoint50. Semantically, multiple pathways can be used to carry the traffic associated with system10. In contrast to other applications that bundle their traffic (i.e., provide a single hole into the firewall), certain implementations of system10can employ two different pathways in the firewall: two pathways for carrying two different types of data.

The objects within video services56are network elements that route or that switch (or that cooperate with each other in order to route or switch) traffic and/or packets in a network environment. As used herein in this Specification, the term ‘network element’ is meant to encompass servers, switches, routers, gateways, bridges, loadbalancers, firewalls, inline service nodes, proxies, processors, modules, or any other suitable device, component, element, or object operable to exchange information in a network environment. This network element may include any suitable hardware, software, components, modules, interfaces, or objects that facilitate the operations thereof. This may be inclusive of appropriate algorithms and communication protocols that allow for the effective exchange (reception and/or transmission) of data or information.

Note that videomail server60may share (or coordinate) certain processing operations between any of the elements of video services56. Using a similar rationale, their respective memory elements may store, maintain, and/or update data in any number of possible manners. In one example implementation, videomail server60can include software to achieve the video processing applications involving the user, as described herein. In other embodiments, these features may be provided externally to any of the aforementioned elements, or included in some other network element to achieve this intended functionality. Alternatively, several elements may include software (or reciprocating software) that can coordinate in order to achieve the operations, as outlined herein. In still other embodiments, any of the devices of the FIGURES may include any suitable algorithms, hardware, software, components, modules, interfaces, or objects that facilitate these switching operations.

In certain instances, videomail60can be provisioned in a different location, or some other functionalities can be provided directly within the videoconferencing platform (e.g., within console element20, camera element14, display12, etc.). This could be the case in scenarios in which console element20has been provisioned with increased intelligence to perform similar tasks, or to manage certain repositories of data for the benefit of the individual user.

FIG. 4is a simplified block diagram illustrating details associated with call signaling and call media. In this particular instance, the call media links are provided in broken lines, whereas the call signaling links are provided as straight-lines. More specifically, call signaling propagates from a set of endpoints74a-bover a broadband network, where these links have a suitable connection at video services56. These links are labeled70a-bin the example ofFIG. 4. Video services56include many of the services identified previously with respect toFIG. 3. Call media between endpoints74a-bpropagate over the broadband network, where these links are identified as72a-b. Endpoints74a-bare simply videoconferencing entities that are leveraging the equipment of system10.

FIGS. 5-7are simplified flow diagrams illustrating certain audio features of the present disclosure. In regards to audio data (inclusive of voice data), in one particular example, speakers16are provisioned as a microphone array, which can be suitably calibrated. Note that in certain consumer applications, the consumer's home system is the variant, which is in contrast to most enterprise systems that have fixed (predictable) office structures. Camera element14can include an array of eight microphones in a particular example, but alternatively any number of microphones can be provisioned to suitably capture audio data. The microphones can be spaced linearly, equidistantly, or logarithmically in order to capture audio data. MicroElectrical-Mechanical System (MEMS) technology can be employed for each microphone in certain implementations of system10. The MEMS microphones represent variations of the condenser microphone design, having a built in analog-to-digital converter (ADC) circuits.

In operation, the audio mechanisms of the videoconferencing architecture can be configured to add a delay to system10in order to ensure that the acoustics function properly. In essence, the videoconferencing architecture does not inherently know the appropriate delay because of the unique domain of the consumer. The delay is generally caused by the television system because each television system is unique (e.g., different consumer brands/models having different delays). For example, there could be a home theater system being used for acoustic purposes. Hence, system10can determine the proper delay, which would be unique for that particular environment. In one particular instance, the delay can be measured, where the echoing effects from the existing speakers are suitably canceled. In one example implementation, the audio system of the present disclosure is configured to measure the delay of a given system by playing music during an initial startup phase. In one particular example, this intelligent delay mechanism can be provided in a given processing element (e.g., within a digital signal processor (DSP), as shown inFIG. 8). Thus, the calibration activities can be accomplished using white noise or music, which may be more pleasant to the end user. Hence, the architecture can also send out a signal (e.g., music, white noise) as a test for measuring the appropriate delay. In certain instances, this function is done automatically without having to prompt the end user.

The architecture can also employ wireless microphone24, which can use a dedicated link in certain implementations. Wireless microphone24can be paired (akin to Bluetooth pairing) such that privacy issues can be suitably addressed. Wireless microphone24can be taken anywhere (e.g., in the room, in the house, etc.) and still provide appropriate audio functions, where multiplexing would occur at console element20for this particular application. Similarly, there could be an incarnation of the same for a given speaker (or the speaker/microphone can be provided together as a mobile unit, which would be portable). The speaker could be similarly used anywhere in the room, in the house, etc. It should be noted that this is not only a convenience issue, but also a performance issue in suitably capturing/delivering audio signals having the proper strength and quality.

Additionally, an embedded watermarking signature can be provided in each of the speakers of the audio system, where the signature can be detected in order to determine an appropriate delay. Note that there is also some additional delay added by display12itself because the clocking mechanism is generally not deterministic. The architecture can dynamically update the delay to account for this issue. Many of these functions can be accomplished by console element20and/or camera element14: both of which can be intelligently configured for performing these function adjustments.

In terms of call answering and video messaging, handset28allows an individual to have the option of taking a voice call instead of answering a videoconferencing call. This is because handset28can have the intelligence to operate purely as a mobile phone. For this reason, handset28can readily be substituted/replaced by various types of smartphones, which could have an application provisioned thereon for controlling the videoconferencing activities. Handset28also affords the ability to be notified (through handset28itself) of an incoming videoconferencing call, with the option of rendering that call on display12. A simple visual alert (e.g., an LED, a vibration, etc.) can be used to indicate that a video message is waiting to be heard/watched.

In operation of a typical videoconferencing call, the call begins much like placing a regular telephone call on a cordless phone. If the recipient of the call answers, the call can begin audio-only, and the audio streams can terminate at handset28by default. At any time during the call, both parties can elect to move to a full video session, which triggers the switching of video streaming and directing call audio to the audiovisual system (via console element20). The audio routing can then be switched freely and on-demand between the audiovisual stack and handset28. In addition, handset28can have two speakers: one for holding close to the ear, and the other for speakerphone operation. Accordingly, the platform can support dynamic routing between audio sources and sinks.

Handset28can support encoding and decoding of the G.711 audio codec for call audio. Additionally, handset28can support the G.722 wideband codec. In one particular example, the streaming protocol can be SRTP. Session keys can be distributed over the existing secure communication channel, extending the existing key management infrastructure for this purpose. The audio input and output stream of console element20can connect a given processor (e.g., a field programmable gate array (FPGA)), where it can be formatted and routed accordingly to the proper destination. Eight channels of audio (from the microphone array inside camera element14) can be formatted into one time division multiplex (TDM) stream before being sent to the audio DSP (ADSP) for further processing. The FPGA can also be responsible for relaying the audio input from the main input, or for passing the decoded audio stream of the remote participants to the television output port.

In regards to the initial provisioning and the first-time setup, the process is relatively quick and simple and, further, can exploit opportunities for automation (where possible). For the initial boot, console element20and handset28can be initiated independently. A welcome screen can initially be shown on each device (such that the user has some assurance that the device is operating normally). For pairing activities, handset28can be configured to generate a secure object (e.g., a nonce) to be sent to console element20via some suitable infrared transmission. The nonce can be used as a password for securing a private WiFi network in certain implementations. The SSID for the network (or a pattern to match) can be predefined. This network can be used for two-way communications between the two devices until it is optionally replaced by an external wireless router. USB is also supported for communicating the data to be used for a suitable pairing. Once connectivity is established, the devices can then authenticate each other.

In terms of audio/video connectivity, video connectivity can be verified by playing an animation on display12(e.g., the HDTV) and, further, by asking for some type of confirmation from the user. The system is able to verify that there is a powered device cabled to the HDMI output. Specifically, in the case of audio, the system can play a test sound using the microphones on handset28and/or the speakers of camera element14(and verify the associated detection). For camera element14, the architecture can simply initiate the camera and set the video routing system to a loopback mode. The user can be prompted to make mechanical adjustments to the camera tilt: aiming it at the desired location. Camera calibration (e.g., such as gain and color balance) can be done at this point, and the user can be prompted to adjust room lighting if necessary. The user can have an opportunity to take a directory photo at this juncture, where this image can be cached locally until the architecture establishes upstream connectivity.

In regards to the speaker interface, there can be a number of APIs used for remote speaker operation. In one particular example, the audio files can be provided in a pulse code modulation (PCM) format. The following list illustrates a number of possible APIs that can be configured in the system:1) set_speaker_volume_level (level): Set the playback volume to be used when playing audio files.2) store_audio_file (file_name, file): Store the audio file by name. (The API may be replaced with a store_file Flash File System API.)3) play_local_audio_file (file_name): Play the given audio file contained in the flash file system.4) play_remote_audio_file (file_name): Stream an audio file stored on the remote host file-system. Audio files that are too large for the camera-board flash can be ‘streamed’ in and, subsequently, played on the fly.5) set_default_audio_file_for_event (event_name, file_name): This API can be used to specify the audio file to be played when a particular event occurs. One such event could be provided as ‘USB-LINK_DOWN.’

Turning to the particular operations ofFIG. 5,FIG. 5is a simplified flowchart82illustrating an overview of a call flow associated with the present disclosure. In this particular example, the call flow involves a first endpoint, a server, and a second endpoint. The endpoints can be associated with videoconferencing equipment for conducting video sessions (e.g., such as that which is depicted byFIG. 1). The server in this particular flow can be any type of server configured to facilitate data flows between the endpoints. For example, the server can be a POP server configured to foster communications between videoconferencing endpoints.

In this particular example, the flow is associated with an Interactive Connectivity Establishment (ICE) protocol, but the present disclosure is not limited to such protocols. The call flow may begin when the user initiates a voice conversation. An INVITE message is sent at [1]. The INVITE message can specify the candidates to be included in this particular call. The server can forward this INVITE along toward its intended recipients at [2]. At a subsequent operation [3], a 183 message is sent by the second endpoint and toward the server. This message is subsequently relayed to the first endpoint, as is being depicted by [4]. 180 ringing messages are exchanged in [5-6], where 200 OK messages are exchanged in [7-8]. ACK messages are exchanged in [9-10]. A re-invite message is propagated toward the server at [11]. The re-invite message is then forwarded to the second endpoint at [12]. Again, 200 OK and ACK messages are exchanged at [13-16]. Multiplex (mux) negotiations occur at [17-18]. 200 OK and ACK messages are exchanged at [19-22]. A real-time protocol (RTP) link is the resultant, where the first and second endpoints can suitably interact and exchange video information.

FIGS. 6A-6Billustrate a simplified flowchart84depicting how a particular call flow can ascend from pure audio to a full audio/video call. This particular flow similarly involves a first endpoint, a second endpoint, and a server. As a general proposition, the architecture can presume that each consumer call would begin as audio-only. Once the call has been established, either endpoint can elect to move to a full video session (e.g., push a particular button on handset28to escalate the call to a full videoconferencing call). Note that in particular implementations, even though the call may begin as audio, the initial INVITE can include both audio and video channels. The call may be initially restricted to audio, as determined by the answering side endpoint.

This particular flow may begin at [1] with an INVITE for particular candidates being sent to the server. The INVITE is relayed to the second endpoint at [2]. 183 messages are exchanged at [3-4]. 180 ringing messages are exchanged at [5-6]. 200 OK messages are exchanged at [7-8], where ACK messages are exchanged at [9-10]. At [11], a re-invite with particular candidates is sent from the first endpoint to the server. This re-invite message is related to the second endpoint at [12]. 200 OK messages are sent at [13-14]. ACK messages are exchanged at [15-16]. Datagram Transport Layer Security (DTLS) and mux negotiations would then ensue.

A re-invite message would then be forwarded to the server, and then communicated to the second endpoint, as shown in [17-18]. 200 OK messages are exchanged at [19-20]. ACK messages are exchanged at [21-22]. A secure real-time transport protocol (SRTP) audio has been effectively established between the endpoints. At [23-24], re-invite messages are communicated to the server and then to the second endpoint. A 200 OK message is communicated at [25-26]. ACK messages are then exchanged at [27-28]. At this juncture, DTLS and mux negotiations for video are established, and the SRTP audio/video link is viable.

FIG. 7is a simplified flowchart86illustrating a call flow associated with downgrading to only audio data. This particular flow presumes that the full video session negotiation has already succeeded, where the first endpoint desires to drop the video media and return to an audio only call. At [1], a re-invite message is sent to the server with a designation for the audio candidate(s). The re-invite is forwarded to the second endpoint at [2]. 200 OK messages are exchanged at [3-4], where the audio candidate is again identified in the message. ACK messages are exchanged at [5-6], where the resultant is an SRTP audio link between the endpoints.

FIG. 8is a simplified block diagram illustrating an audio subsystem100associated with the present disclosure. In a particular example, audio subsystem100includes an FPGA102, along with several audio DSPs104,106. FPGA102can provide an audio multiplexing functionality for audio subsystem100. Also, audio subsystem100includes a CPU110, which is coupled to a peripheral component interconnect express (PCIe) bridge. Operationally, the DSPs can be used to perform audio beam-forming and direction finding, acoustic echo cancellation, and audio compression. In a particular implementation, audio subsystem100is provisioned within console element20to achieve the audio features detailed herein.

Logistically, audio DSP104is connected to the host CPU110over a host parallel interface (HPI) bus in this particular implementation. Audio DSP106is connected to the host CPU110over the PCIe bus. The two DSPs can be connected over a serial high-speed bus. The decoded output audio of the remote participant can be sent to the HDMI out (to the television or the A/V receiver) or encoded (e.g., in G.711 or G.722) before being encapsulated in an IP package and sent to handset28(e.g., through a Wi-Fi network link). Again, routing priorities can be set such that if videoconferencing is not active, or if handset28is active and working in a private mode, the television output can relay audio/video from one of the HDMI input sources.

In a particular implementation, camera element14includes an eight-microphone linear array for beam forming and direction finding. The audio features can apply to several simultaneous speakers and, further, be performed by audio DSPs within console element20. In a particular arrangement, the microphones are tight tolerance Omni-directional digital MEMS. Additionally, there can be four audio codecs that are used to convert the digital MEMS output into I2S format in order to interface with the HDMI transmitter integrated circuit. I2S, also known as Inter-IC Sound, Integrated Interchip Sound, or IIS, is an electrical serial bus interface used for connecting digital audio devices together.

In operation, the audio input and output stream of console element20can connect to FPGA102, where it can be formatted and routed to its proper destination. FPGA102may also be responsible for relaying the audio input from the main input, or for passing the decoded audio stream of the remote participants to the television output port. In a particular implementation, the video FPGA (VFPGA) has three audio input signals and four audio output signals. It is imperative to note that the arrangement ofFIG. 8is simply being used for discussion purposes, where various alternatives and modifications to such an arrangement would be within the scope of the present disclosure. Along similar lines, the processing and memory allocations ofFIG. 8may be shared by other components, consolidated, or otherwise distributed across different devices. Such modifications may be based on particular videoconferencing needs, specific environments, etc.

Note that in certain example implementations, the audio processing functions outlined herein may be implemented by logic encoded in one or more tangible media (e.g., embedded logic provided in an application specific integrated circuit [ASIC], digital signal processor [DSP] instructions, software [potentially inclusive of object code and source code] to be executed by a processor, or any other similar machine, etc.). In some of these instances, a memory element [as shown inFIG. 2] can store data used for the audio enhancement operations described herein (e.g., involving delay, watermarking, calibration, etc.). This includes the memory element being able to store software, logic, code, or processor instructions that are executed to carry out the activities described in this Specification. A processor can execute any type of instructions associated with the data to achieve the operations detailed herein in this Specification. In one example, the processor [as shown inFIG. 2] could transform an element or an article (e.g., data) from one state or thing to another state or thing. In another example, the audio enhancement activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software/computer instructions executed by a processor) and the elements identified herein could be some type of a programmable processor, programmable digital logic (e.g., FPGA, an erasable programmable read only memory (EPROM), an electrically erasable programmable ROM (EEPROM)) or an ASIC that includes digital logic, software, code, electronic instructions, or any suitable combination thereof.

Note that the equipment ofFIGS. 2 and 8may share (or coordinate) certain processing operations. Using a similar rationale, their respective memory elements may store, maintain, and/or update data in any number of possible manners. In a general sense, the arrangements depicted in the preceding FIGURES may be more logical in their representations, whereas a physical architecture may include various permutations/combinations/hybrids of these elements. In one example implementation, console element20and/or audio subsystem100include software (e.g., as part of, or in cooperation with, the processing elements discussed (such as the DSPs, the FPGAs, etc.) to achieve the audio enhancement operations, as outlined herein in this document. In other embodiments, these features may be provided externally to any of the aforementioned elements (e.g., included in camera element14), or included in some other device to achieve these functionalities. Alternatively, several elements may include software (or reciprocating software) that can coordinate in order to achieve the operations, as outlined herein. In still other embodiments, any of the devices of the FIGURES may include any suitable algorithms, hardware, software, components, modules, interfaces, or objects that facilitate these video enhancement operations.

All of the aforementioned devices may further keep information in any suitable memory element (e.g., random access memory (RAM), ROM, EPROM, EEPROM, ASIC, etc.), software, hardware, or in any other suitable component, device, element, or object where appropriate and based on particular needs. Any of the memory items discussed herein (e.g., database, table, key, queue, etc.) should be construed as being encompassed within the broad term ‘memory element.’ Similarly, any of the potential processing elements, modules, and machines described in this Specification should be construed as being encompassed within the broad term ‘processor.’ Console element20, audio subsystem100, and/or camera element14can also include suitable interfaces for receiving, transmitting, and/or otherwise communicating data or information in a network environment.

Although the present disclosure has been described in detail with reference to particular arrangements and configurations, these example configurations and arrangements may be changed significantly without departing from the scope of the present disclosure. For example, although the present disclosure has been described with reference to particular communication exchanges involving certain server components, system10may be applicable to other protocols and arrangements (e.g., those involving any type of videoconferencing scenarios). Additionally, although camera element14has been described as being mounted in a particular fashion, camera element14could be mounted in any suitable manner in order to suitably capture video images. Other configurations could include suitable wall mountings, aisle mountings, furniture mountings, cabinet mountings, upright (standing) assemblies, etc., or arrangements in which cameras would be appropriately spaced or positioned to perform its functions.

Furthermore, the users described herein are simply individuals within the proximity, or within the field of view, of display12. Audience members can be persons engaged in a video conference involving other individuals at a remote site. Audience members can be associated with corporate scenarios, consumer scenarios, residential scenarios, etc. or associated with any other suitable environment to which system10may be applicable.

Additionally, system10can involve different types of counterparties, where there can be asymmetry in the technologies being employed by the individuals. For example, one user may be using a laptop, while another user is using the architecture of system10. Similarly, a smartphone could be used as one individual endpoint, while another user continues to use the architecture of system10. Also, Webcams can readily be used in conjunction with system10. Along similar lines, multiparty calls can readily be achieved using the teachings of the present disclosure. Moreover, although system10has been illustrated with reference to particular elements and operations that facilitate the communication process, these elements and operations may be replaced by any suitable architecture or process that achieves the intended functionality of system10.