Adjusting audio volume and quality of near end and far end talkers

An orchestrator associated with a collaboration application client executed by a near end device dynamically adapts volume level and/or other audio quality parameters to deliver a consistent voice experience to a collaboration participant. The orchestrator is informed by multiple machine learning engines collecting and analyzing inputs from one or more existing sensor-based functions embedded in the near end device. The orchestrator determine an audio configuration of the device and audio preferences of the user. Identities of far end participants are determined and their profiles are mapped against the user volume preferences. The orchestrator functions as an informing agent to the audio signal processing engine of the device, managing speaker output volume and microphone gain settings, based upon the machine learning engines and the sensor-based functions. The sensor based functions may detect proximity, head pose, gaze point, eye position, facial identities, mood, and so forth.

TECHNICAL FIELD

The present disclosure relates to collaboration applications and, more specifically, audio quality experienced by users of collaboration applications.

BACKGROUND

Information handling systems are frequently employed to engage in distanced collaboration sessions in which one or more attendees at each of two or more distinct locations invoke a collaboration application program such as Zoom, Microsoft Teams, or the like to establish a shared communication link, which typically includes an audio component and frequently including a video component.

The sound level and quality experienced between different collaboration software applications varies noticeably. The factors affecting the quality of the audio experience include: variations in the devices used to join the collaboration application, different room settings for the one or more far end talkers as well as the near end user, variations in talker volume, pitch, etc., varying number of attendees, different and often changing positions of talkers relative to microphone or speaker, different volume level management techniques employed by different collaboration applications. Users desire a smooth and level volume from all attendees irrespective of these and other variables.

SUMMARY

In accordance with the teachings of the present disclosure, problems associated with audio quality experienced by users of collaboration applications is reduced or eliminated by one or more disclosed methods of adjusting and enhancing audio volume and/or one or more other audio signal parameters. A disclosed software module referred to herein as an orchestrator, combines one or more machine learning functions, modules, or engines with one or more sensor-based functions to automatically detect, identify, and learn voice characteristics of attendees and their corresponding preferences and to uniquely adapt volume level and other audio quality parameters to deliver a consistent voice experience to each user. The orchestrator serves as the informing agent to the audio signal processing engine of the user's device based upon the combined inputs from the machine learning engines and the sensor-based functions. The orchestrator beneficially utilizes one or more pre-existing system capabilities including, as non-limiting examples, capabilities for proximity detection, head tracking, eye gaze, facial recognition and facial identification, and so forth.

In at least one embodiment, the orchestrator is configured to access or generate profiles of one or more attendees of the collaboration session. Disclosed systems may access or employ one or more machine learning engines for aggregating attendee profile information and mapping the information against user volume level preferences. This intelligence may be augmented with information from client-based sensors and sensor functions embedded in the user's device, including, without limitation, proximity sensors, eye track sensors, head angle sensors, and so forth. The aggregate of all learned and sensed intelligence drive a common layer of control in the form of the orchestration module managing speaker output volume and microphone gain settings.

The orchestrator may support two or more optimization phases, each of which may be associated with particular functions. As a non-limiting example, an exemplary orchestrator may support a start phase, a detect phase, a discovery phase, a profile phase, and an action phase.

The orchestrator application may monitor inputs from one or more embedded sensors and sensor functions within the user's device and one or more machine learning engines to seamlessly and dynamically adjust one or more audio parameters including but not limited to volume level. In this manner, the orchestrator presents the user with smooth and level audio despite variations in one or more audio-relevant parameters. In at least some embodiments, the types of variation that the orchestrator may encounter and combat include, without limitation, loud and soft spoken speakers, variations in the number of participants, variations in the acoustic parameters of the room or environment from which each participant joins the collaboration session, variations in the position of the user with respect to the applicable microphone and audio speaker, variations in level management techniques between or among two or more collaboration applications.

Subject matter included herein discloses an orchestrator software module, associated with a collaboration application client executed by a near end device, which dynamically characterizes near and far end volumes levels and dynamically adapts near end volume level and/or other audio quality parameters to deliver a consistent voice experience to a collaboration participant. The orchestrator is informed by multiple machine learning engines collecting and analyzing inputs from one or more existing sensor-based functions embedded in the near end device. The orchestrator determine an audio configuration of the device and audio preferences of the user. Identities of far end participants are determined and their profiles are mapped against the user volume preferences. The orchestrator functions as an informing agent to the audio signal processing engine of the device, managing speaker output volume and microphone gain settings, based upon the machine learning engines and the sensor-based functions. The sensor based functions may detect proximity, head pose, gaze point, eye position, facial identities, mood, and so forth.

Technical advantages of the present disclosure may be apparent to those of ordinary skill in the art in view of the following specification, claims, and drawings.

DETAILED DESCRIPTION

Preferred embodiments and their advantages are best understood by reference toFIGS.1-7, wherein like numbers are used to indicate like and corresponding parts.

In this disclosure, the term “information handling resource” may broadly refer to any component system, device or apparatus of an information handling system, including without limitation processors, buses, memories, input-output devices and/or interfaces, storage resources, network interfaces, motherboards, electro-mechanical devices (e.g., fans), displays, and power supplies.

Referring now to the drawings,FIG.1andFIG.2illustrate examples of undesirable audio inconsistency and variability associated with collaboration applications. The illustrated examples are representative of audio issues addressed by an information handling resource disclose and referred to herein as the collaboration session orchestrator or, more simply, the orchestrator.FIG.1illustrates an example collaboration session100including a near end participant101, also referred to herein as near end user101, seated in front of a desktop computer113, and two far end participants102-1and102-2, also referred to herein as far end talkers or far end speakers. As depicted inFIG.1, first far end participant102-1has a comparatively loud speaking voice, as conveyed by the comparatively large amplitude of the representative sound wave105-1, while second far end participant102-2has a comparatively soft or quiet speaking voice, as conveyed by the comparatively small amplitude of the representative sound wave105-2.FIG.1further illustrates the near end user's optimal volume levels107-1and107-2for these two far end speakers. In this context, optimal volume levels107represent volume levels that produce corresponding audible outputs109that are, from the perspective of near end user101, level or uniform across both far end speakers and optimal with respect to the near end user's personal preferences.

FIG.2illustrates near end user101speaking from a seated position in front of a desktop computer device113and in close proximity to an external microphone111during a collaboration session.FIG.2further illustrates near end user101speaking from a standing position115at some distance from a mobile computer117, e.g., a laptop or surface device, during a different collaboration session or at a different time of the same collaboration session. Under the illustrated conditions, external microphone111and a built in microphone (not explicitly depicted) in mobile computer117respectively generate first and second audio signals119-1and119-2corresponding to the near end users voice.FIG.2emphasizes that the amplitude of first audio signal119-1is considerably different and larger than the amplitude of second audio signal119-2and it will be readily appreciated that this variation will be detectable to far end participants.

For the sake of clarity and brevity, the sources of audio quality variability emphasized inFIG.1andFIG.2are merely examples and those of ordinary skill in the field of collaboration solutions will recognize that there are many other sources of variability. An orchestrator disclosed herein and described in more detail below monitors one or more sensor-based and machine-derived inputs to detect and compensate for all such sources of audio quality variability. The orchestrator beneficially utilizes intelligence functionality including, without limitation, proximity detection, head tracking, eye gaze, facial recognition, etc., that is substantially or entirely pre-existing in the devices typically used to by collaboration session participants, but as the informing agent for a device's audio signal processing resources, the specific combinations of sensor-based and machine learning inputs assessed by the orchestrator are unique and unprecedented.

Referring now toFIG.3, resources employed in an exemplary collaboration session100are illustrated. For the sake of clarity, the collaboration session100illustrated inFIG.3has just three participants or, more specifically, three participant locations, each of which includes one or more participants, but those of ordinary skill will appreciate that the concepts disclosed herein are readily scalable and that the number of participants and participant locations can be considerably larger than the three locations ofFIG.3.

The illustrated collaboration session100includes a near end collaboration device121-1associated with a near end user101located at near end location171, a first far end collaboration device122-1associated with a first far end user102-1located at a first far end location171-1, and a second far end collaboration device122-2associated with a group of four far end users102-2through102-5located at a second far end location172-2. Collaboration devices121and122are information handling systems that include or have access to capabilities and hardware resources for generating, encoding, and transmitting audio signals and receiving, decoding, and rendering audio and video signals. Collaboration devices121and122are typically web-capable devices configured to support point-to-point and multipoint audio-visual sessions in compliance with one or more standards and/or protocols for networked communication of audio-video content including, as two pervasive but non-limiting examples, H.323 and Session Initiation Protocol (SIP). Collaboration devices121and122may be implemented with any of a variety of information handling system types including, as non-limiting examples, smart phones, tablets, laptop and desktop computers, hybrid devices including Microsoft Surfaces devices, gaming controllers, docking stations, dedicated conference phones and audio/video bars, in combination with one or more large screen monitors, and so forth. Many information handling systems suitable for use as a collaboration device include or support functions, features, and resources, discussed in more detail below with respect toFIG.4, that are utilized for audio quality leveling as disclosed herein.

In at least one embodiment, each collaboration device121and122executes a collaboration client (not explicitly depicted inFIG.3) to connect with a virtualized collaboration server162associated with a collaboration service provider. Among many other features and functions, including call control and security, collaboration sever162includes a multipoint control unit (MCU)231to receive audio and video content signals from each collaboration device and return a mixed audio signal one or more mixed and/or switched video signals to each collaboration device so that each participant hears a composite audio signal of all speakers and sees one or more of the other participants including the active speaker.

As depicted inFIG.3, collaboration device121is communicatively coupled to one or more functions, modules, engines, or resources generically referred to herein as machine learning engine(s)181. In at least some embodiments, volume level control information may be generated based on the usage parameters, including as a non-limiting example, volume changes performed by the user during other collaboration sessions, using artificial intelligence and/or machine learning algorithms, such as supervised, unsupervised, or reinforcement training algorithms, to analyze the usage parameters and determine volume level control information that will enhance system performance. A machine learning engine may use algorithms to identify relationships between user-initiated changes in the user's volume level and one or more other parameters, conditions, or states relevant to the collaboration session. To illustrate, machine learning engines may determine, perhaps not surprisingly, a correlation between the volume level selected by the user and any one or more of the following illustrative factors or parameters: the user's proximity, gaze point, head pose etc. relative to the user's device, the number of attendees at one or more of far end locations, the type of room or environment associated with the location of the user or one or more of the far end locations; whether the user employed a headset during the session and/or participated via smart phone; bandwidth and/or noise parameters associated with the user's network connection, the time of day, day of week, room temperature and humidity, and so forth, and determine volume level control information that will improve the uniformity of the volume experienced by the user. In some embodiments, a rule-based engine may be used alone or in combination with other algorithms for determining volume level control information. After any volume level control information is determined, the information may be distributed to one or more of the users.

Volume level control information may be generated through the application of optimization algorithms, such as machine learning and/or artificial intelligence algorithms. The optimization algorithms may make use of usage parameters and corresponding volume level control information, previously generated and/or received from other information handling systems, in generating volume level control information based on the received usage parameters. For example, machine learning and/or artificial intelligence algorithms may be used to analyze combinations of stored usage and volume level control information to determine ways in which adjustments have caused improvements in system performance, which may be indicated by changes in usage parameters following adjustment of volume level control information.

Audio leveling described herein does not impose any requirements on the collaboration service, but may use, access, or otherwise leverage services and features that are provided. As an example, if the service displays the active speaker, this feature might be used in conjunction with facial recognition capabilities resident on at least some of the collaboration devices to identify the participants on a call and, perhaps more significantly with respect to the audio quality issues addressed by disclosed subject matter, identify the talking participants.

Referring now toFIG.4, a block diagram of an information handling system200suitable for use as the collaboration device221illustrated inFIG.3is presented. Although the information handling system200illustrated inFIG.4includes elements that may be associated with a laptop or desktop computer, disclosed audio leveling features may be beneficially included in other types of information handling systems and those of ordinary skill in the field of electronic devices will readily appreciate that the depicted system is exemplary and that other devices, not explicitly illustrated inFIG.4, including smart phones, tablets, hybrid devices, and dedicated video/conferencing devices. It will be further appreciated that, for the sake of clarity and brevity, many elements and components of information handling system200have been omitted from the depiction inFIG.4.

The information handling system200illustrated inFIG.4includes a general purpose processor or central processing unit (CPU)201communicatively coupled to various peripheral devices generically referred to herein as information handling resources. The information handling resources illustrated inFIG.4include a system memory202suitable for storing data (not explicitly depicted inFIG.1) intended for and/or generated by CPU201as well as computer executable instructions, sometimes referred to as programs, applications, and the like, for performing specific tasks and functions.

The programs residing in the system memory202illustrated inFIG.4include an operating system203which manages system resources and provides a functional platform for CPU201to execute application programs. The applications programs residing in the system memory202illustrated inFIG.4include a collaboration application client referred to herein as collaboration application231and an orchestrator program251for implementing audio leveling in conjunction with the collaboration application231. For the sake of clarity, many other programs executed by CPU201, all or portions of which may be stored in system memory202, are omitted fromFIG.4.

In at least one embodiment, orchestrator application204monitors inputs from one or more sensors and one or more machine learning engines that may be germane to the audio quality experienced by a participant of a collaboration session accessed via collaboration application231to seamlessly and dynamically adjust one or more audio parameters including but not limited to volume. In this manner, orchestrator251presents the user with smooth and level audio despite variations in one or more audio-relevant parameters. In at least some embodiments, the types of variation that orchestrator251may encounter and combat include, without limitation, loud and soft spoken speakers, variations in the number of participants, variations in the acoustic parameters of the room or environment from which each participant joins the collaboration session, variations in the position of the user with respect to the applicable microphone and audio speaker, variations in level management techniques between or among two or more collaboration applications.

The information handling system200ofFIG.4further includes a graphics module211to process video information and render video on a display device such as the liquid crystal display (LCD)212and a chipset220to communicatively couple various peripheral devices to CPU201. A network interface card (NIC)213supports Ethernet or another suitable broadband network connection. A baseboard management controller (BMC)214facilitates and supports external management of information handling system200. The peripheral devices coupled to CPU201via chip set260include storage resources261, camera263, radio/transceiver resources264-267supporting various wireless communication transports and protocols, an audio codec271coupled to a microphone272and speakers273, and an IR transceiver281.

The information handling system200illustrated inFIG.4includes one or more sensors and/or sensed functions which may be utilized by orchestrator251. The sensed functions illustrated inFIG.4include a proximity detector282, eye tracker285, facial recognition module287, and an acoustic resource291. Eye tracker285may support monitoring and determination of eye position, gaze direction, and head pose, as non-limiting examples. Acoustic modules291may be used in conjunction with microphone272to measure the spectral content and other characteristics of a talker and determine the echo characteristics of a room or environment by, for example, determining an RT60 value or another reverberation time parameter for a room.

Referring now toFIG.5, a flow diagram illustrates an exemplary audio parameter control method400performed by orchestrator251in conjunction with one or more machine learning engines402, two of which are illustrated inFIGS.5(402-1and402-2). Although method400encompasses the control of any suitable audio signal parameter, for the sake of brevity and clarity, the illustrated method is explained with respect to volume as the most readily familiar audio parameter.

As illustrated inFIG.5, method400includes detecting (410) an audio configuration of the user's collaboration device, i.e., the information handling system resources employed by the user to participate in the collaboration session. Referring briefly toFIG.6, various exemplary collaboration device configurations are illustrated. The illustrated collaboration devices configurations include a first configuration301comprising a stand-alone laptop or desktop computer, a second configuration602comprising a, a laptop/desktop computer in combination with a Bluetooth headset, a third configuration603comprising a laptop computer coupled to a docking station and sound bar, a fourth configuration comprising a stand-alone smart phone, tablet, other suitable mobile device, a fifth configuration605comprising a mobile device in combination with a Bluetooth headset, and a six configuration, suitable for a conference room or the like, comprising a dedicated tabletop microphone array in combination with a sound bar and large screen flat panel display. Each configuration features a different path for rendering audio and capturing microphone input. In at least some embodiments, orchestrator251may capture the user device configuration based via APIs (e.g., MIPI, ACPI and other standard methods).

Method400includes profiling (412) the users characteristics (e.g., face ID, spectral content of voice, etc.) and observing and learning (414) the user's volume preferences. Based on the user device configuration and the user's profile and volume preferences, the orchestrator applies (420).

The method400illustrated inFIG.5further includes ongoing monitoring (422) of sensor based indicators of the user's presence and state within the collaboration session and with respect to the audio-significant elements of the user's device configuration (e.g., the user's position with respect to the applicable microphone and speakers as determined by one or more existing capabilities of intelligence in the user device (e.g., proximity detection, head tracking, eye gaze, face ID, etc.)

One or more machine learning engines402may aggregate (424) attendee profiles, map (428) them against preferred user volume, combined with the user profiling obtained by monitoring (422) embedded client based sensors (like proximity sensing and eye track/head angle). Method400aggregates these inputs via a common layer of control through an orchestration layer to augment (430) output volume and microphone gain settings.

Method400can function independent of the collaboration software and learns/adapts to the uniqueness of the hardware configuration and user environment. The audio signals captured by the microphones or rendered on the loudspeakers are modified via client-based processing that is informed by the machine learning engines' outputs. Method400will also learn which collaboration app is being used and updates preferences and volume adjustments associated with each app.

Referring now toFIGS.7A and7B, aspects of the method400illustrated inFIG.6are illustrated in sequence diagram700. Sequence diagram700identifies four primary phases of interaction between and among the illustrated resources. During detection phase710, orchestrator251detects the user's configuration which may include a dock712, system audio714, camera716, and eye tracker718.

During discovery phase720, orchestrator251accesses machine learning engine181to establish a baseline volume and microphone gain settings via volume manager726and microphone sensing728respectively. The baseline volume setting may be augmented based on inputs from sensors and sensor functions embedded in the user device. Such inputs may include, as non-limiting examples, the user's proximity to the user device (via proximity sensing722), the user's head pose, gaze point, and/or eye position via eye tracker (724).

During a profiling phase (730), attendees are identified and their profiles aggregated by machine learning engine181and augmented based on room type732and/or facial recognition734.

In action phase740, orchestrator251controls the volume level of the user's configuration based on a composite of the information provided via the various machine learning engines. In at least one embodiment, the logic employs a “do no harm” approach in conjunction with a composite of the results from the multiple machine learning engines, upon which orchestrator251can take an action to control the ‘knob” such as the volume for the applicable user.

Although the present disclosure has been described in detail, it should be understood that various changes, substitutions, and alterations can be made hereto without departing from the spirit and the scope of the disclosure as defined by the appended claims.