SYSTEM AND METHOD FOR USING PREDICTIVE ANALYSIS TO GENERATE A HIERARCHICAL GRAPHICAL LAYOUT

Methods and systems for using predictive analysis to generate a hierarchical layout of participants in a video conference on display devices are provided herein. In some embodiments, a method for using predictive analysis to generate a hierarchical layout of participants in a video conference on display devices comprises receiving a request to set up a video conference among three or more participants predicting, based on historical data, a dominant participant of the three or more participants in the video conference; and displaying a hierarchical layout of the participants on display devices of the two or more remaining subordinate speakers, who are not predicted to be the dominant speaker, with the predicted dominant participant displayed more prominently than the remaining three or more participants.

BACKGROUND

Field

Embodiments of the present invention generally relate to video conferencing, and more specifically, to a method and system for using predictive analysis to generate a hierarchical graphical layout of participants in a video conferencing session.

Description of the Related Art

Oftentimes during a video conferencing call involving multiple participants, the participants are displayed on each user device display in an illogical order. For example, a participant who has little to contribute to the video conference may be displayed first, while a participant displayed in the middle or end of the group of participants may have a more active role in the call. This makes participants search for a current speaker on the display, leading to a poor user experience.

Therefore, there is a need in the art for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session.

SUMMARY

In some embodiments, a method for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session includes receiving a request to set up a video conference among three or more participants; predicting, based on historical data, a dominant speaker of the three or more participants in the video conference; and displaying a hierarchical layout of the participants on display devices of the three or more participants with the predicted dominant speaker displayed more prominently than the remaining three or more participants

In some embodiments, a system for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session includes at least one storage device storing processor-executable instructions which, when executed by the at least one processor, perform the method for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session.

In some embodiments, a computer readable medium for storing computer instructions that, when executed by at least one processor causes the at least one processor to perform the method for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session.

DETAILED DESCRIPTION

Embodiments of the present invention generally relate to a method and system for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session. When a video conference involving three or more participants is to begin, historical data regarding the three or more participants in analyzed in order to predict which of the participants is likely to be the dominant speaker in the conference call. Historical data includes at least determining which of the three or more participants spoke most in previous interactions, specifically prior voice or video communications. Additional considerations in determining the dominant speaker include which participant organized the conference call, the topic of the conference call, the title or position of each participant, the subject matter expertise of each participant as it relates to the topic of the call, a geographic location, and/or an office or branch location of the participant, and the like. The three or more participants are ranked based on who is predicted to be the dominant speaker, who is predicted to be the second most dominant speaker, who is predicted to be third most dominant speaker, and so on. As used herein, the participants who are not predicted to be the dominant speaker are referred to as subordinate speakers. The predicted dominant speaker is displayed in a conference call display area more prominently than the subordinate speakers on the display devices of the subordinate speakers. The subordinate speakers are displayed on the display device of the dominant speaker. In some embodiments, the subordinate speakers are displayed on the display device of the dominant speaker with the predicted second most dominant speaker displayed more prominently than the remaining subordinate speakers. In some embodiments, the display of the participants is adjusted if the talk time of the predicted dominant speaker drops below a predefined threshold. In some embodiments, additional bandwidth is provided to the display device of the predicted dominant speaker. In some embodiments, the total bandwidth may be divided among the display devices of a predefined number of predicted top speakers.

In the following description, the terms VOIP system, VOIP telephony system, IP system and IP telephony system are all intended to refer to a system that connects callers and that delivers data, text and video communications using Internet protocol data communications. Those of ordinary skill in the art will recognize that embodiments of the present invention are not limited to use with IP telephony systems and may also be used in other systems.

As illustrated inFIG. 1, a communications environment100is provided to facilitate IP enhanced communications. An IP telephony system120enables connection of telephone calls between its own customers and other parties via data communications that pass over a data network110. The data network110is commonly the Internet, although the IP telephony system120may also make use of private data networks. The IP telephony system120is connected to the Internet110. In addition, the IP telephony system120is connected to a publicly switched telephone network (PSTN)130via a gateway122. The PSTN130may also be directly coupled to the Internet110through one of its own internal gateways (not shown). Thus, communications may pass back and forth between the IP telephony system120and the PSTN130through the Internet110via a gateway maintained within the PSTN130.

The gateway122allows users and devices that are connected to the PSTN130to connect with users and devices that are reachable through the IP telephony system120, and vice versa. In some instances, the gateway122would be a part of the IP telephony system120. In other instances, the gateway122could be maintained by a third party.

Customers of the IP telephony system120can place and receive telephone calls using an IP telephone108that is connected to the Internet110. Such an IP telephone108could be connected to an Internet service provider via a wired connection or via a wireless router. In some instances, the IP telephone108could utilize a packet-switched network of a cellular telephone system to access the Internet110.

Alternatively, a customer could utilize an analog telephone102which is connected to the Internet110via a telephone adapter104. The telephone adapter104converts analog signals from the telephone102into data signals that pass over the Internet110, and vice versa. Analog telephone devices include but are not limited to standard telephones and document imaging devices such as facsimile machines. A configuration using a telephone adapter104is common where the analog telephone102is located in a residence or business. Other configurations are also possible where multiple analog telephones share access through the same IP adaptor. In those situations, all analog telephones could share the same telephone number, or multiple communication lines (e.g., additional telephone numbers) may provisioned by the IP telephony system120.

In addition, a customer could utilize a soft-phone client running on a computer106or a television109to place and receive IP based telephone calls, and to access other IP telephony systems (not shown). The computer106may be a personal computer (PC), a tablet device, a gaming system, and the like. In some instances, the soft-phone client could be assigned its own telephone number. In other instances, the soft-phone client could be associated with a telephone number that is also assigned to an IP telephone108, or to a telephone adaptor104that is connected one or more analog telephones102.

Users of the IP telephony system120are able to access the service from virtually any location where they can connect to the Internet110. Thus, a customer could register with an IP telephony system provider in the U.S., and that customer could then use an IP telephone108located in a country outside the U.S. to access the services. Likewise, the customer could also utilize a computer outside the U.S. that is running a soft-phone client to access the IP telephony system120.

A third party using an analog telephone132which is connected to the PSTN130may call a customer of the IP telephony system120. In this instance, the call is initially connected from the analog telephone132to the PSTN130, and then from the PSTN130, through the gateway122to the IP telephony system120. The IP telephony system120then routes the call to the customer's IP telephony device. A third party using a cellular telephone134could also place a call to an IP telephony system customer, and the connection would be established in a similar manner, although the first link would involve communications between the cellular telephone134and a cellular telephone network. For purposes of this explanation, the cellular telephone network is considered part of the PSTN130.

In the following description, references will be made to an “IP telephony device.” This term is used to refer to any type of device which is capable of interacting with an IP telephony system to complete an audio or video telephone call or to send and receive text messages, and other forms of communications. An IP telephony device could be an IP telephone, a computer running IP telephony software, a telephone adapter which is itself connected to a normal analog telephone, or some other type of device capable of communicating via data packets. An IP telephony device could also be a cellular telephone or a portable computing device that runs a software application that enables the device to act as an IP telephone. Thus, a single device might be capable of operating as both a cellular telephone that can facilitate voice-based session calls, and an IP telephone that can facilitate data-based session calls.

The following description will also refer to a mobile telephony device. The term “mobile telephony device” is intended to encompass multiple different types of devices. In some instances, a mobile telephony device could be a cellular telephone. In other instances, a mobile telephony device may be a mobile computing device, such as the APPLE IPHONE, that includes both cellular telephone capabilities and a wireless data transceiver that can establish a wireless data connection to a data network. Such a mobile computing device could run appropriate application software to conduct VoIP telephone calls via a wireless data connection. Thus, a mobile computing device, such as an APPLE IPHONE, a RIM BLACKBERRY or a comparable device running GOOGLE ANDROID operating system could be a mobile telephony device.

In still other instances, a mobile telephony device may be a device that is not traditionally used as a telephony device, but which includes a wireless data transceiver that can establish a wireless data connection to a data network. Examples of such devices include the APPLE IPOD TOUCH and the IPAD. Such a device may act as a mobile telephony device once it is configured with appropriate application software.

FIG. 1illustrates that a mobile computing device with cellular capabilities136A(e.g., a smartphone) is capable of establishing a first wireless data connection A with a first wireless access point140, such as a WiFi or WiMax router. The first wireless access point140is coupled to the Internet110. Thus, the mobile computing device136Acan establish a VOIP telephone call with the IP telephony system120via a path through the Internet110and the first wireless access point140.

FIG. 1also illustrates that the mobile computing device136Acan establish a second wireless data connection B with a second wireless access point142that is also coupled to the Internet110. Further, the mobile computing device136Acan establish either a third wireless data connection C via a packet-switch network provided by a cellular service provider130using its cellular telephone capabilities, or establish a voice-based session telephone call via a circuit-switched network provided by a cellular service provider130. The mobile computing device136Acould also establish a VoIP telephone call with the IP telephony system120via the second wireless connection B or the third wireless connection C.

Although not illustrated inFIG. 1, the mobile computing device136Amay be capable of establishing a wireless data connection to a data network, such as the Internet110, via alternate means. For example, the mobile computing device136Amight link to some other type of wireless interface using an alternate communication protocol, such as the WIMAX standard. In the embodiments described above, a device may act as a mobile telephony device once it is configured with appropriate application software that may be downloaded from an app distribution platform144. For example, mobile computing device136Amay download a VOIP mobile app from app distribution platform144and install the VOIP mobile app on mobile computing device136A.

Similarly, mobile computing device with cellular capabilities1368may also be coupled to internet110and/or cellular service provider130. In some embodiments, mobile computing device1368may be connected to internet110via a WIFI or WIMAX connection, and the like, and can also establish a VOIP telephone calls with the IP telephony system120similar to mobile computing device136A. In embodiments of the present invention, communications environment100may be used to establish voice based or data-based telecommunications sessions between mobile computing device136Aand mobile computing device1368, depending on various criteria associated with each of the mobile computing devices, as will be described below in more detail.

While the embodiments discussed herein include a centralized identification server that provides recipient information, additional embodiments include a decentralized system, where the end-user devices have the ability to communicate participant information to each other directly.

FIG. 2depicts a block diagram of a system200for using predictive analysis to generate a hierarchical graphical layout during a video conferencing session, according to one or more embodiments. The system200comprises three or more user devices2021,2022, . . .202n(collectively referred to as user device202) and a conference server204communicatively coupled via networks206. In some embodiments, the user device202may be a mobile computing device (e.g.,136A) associated with a user, and conference server204may be a server maintained and operated by IP telephony system120described above inFIG. 1.

Each user device202may comprise a Central Processing Unit (CPU)208, support circuits210, a display212, a video camera214, and a memory216. The CPU208may comprise one or more commercially available microprocessors or microcontrollers that facilitate data processing and storage. The various support circuits210facilitate the operation of the CPU208and include one or more clock circuits, power supplies, cache, input/output device and circuits, and the like. The memory216comprises at least one of Read Only Memory (ROM), Random Access Memory (RAM), disk drive storage, optical storage, removable storage and/or the like. In some embodiments, the memory216comprises an operating system218, a telecommunications module220, and a display module222.

The operating system (OS)218generally manages various computer resources (e.g., network resources, file processors, and/or the like). The operating system218is configured to execute operations on one or more hardware and/or software modules, such as Network Interface Cards (NICs), hard disks, virtualization layers, firewalls and/or the like. Examples of the operating system218may include, but are not limited to, various versions of LINUX, MAC OSX, BSD, UNIX, MICROSOFT WINDOWS,10S, ANDROID and the like.

In some embodiments, the conference server204includes a prediction system230and a conference coordination system248. In some embodiments, the conference coordination system248may be a separate entity that provides conference coordination services to the conference server204. Conference coordination services may include sending conference invitations, collecting participant responses, gathering participant information, coordinating conference call setup among participant devices, monitoring signaling from participant devices, and the like. The prediction system230comprises a Central Processing Unit (CPU)232, support circuits234, and memory236. The CPU232may comprise one or more commercially available microprocessors or microcontrollers that facilitate data processing and storage. The various support circuits234facilitate the operation of the CPU232and include one or more clock circuits, power supplies, cache, input/output circuits, and the like. The memory236comprises at least one of Read Only Memory (ROM), Random Access Memory (RAM), disk drive storage, optical storage, removable storage and/or the like.

In some embodiments, the memory236comprises an operating system238, a call history analyzer240, a conference call database242, a participant database244, a current call monitor246that temporarily stores in a cache, the host of the current call, the topic272of the conference call, the participants274, and a total talk time276for each participant274, as well as other metrics related to the current conference call. The call database242maintains information associated with previous calls250, both voice calls and video conference calls. Each call250includes a plurality of participants252who participated in the call250. Each participant252includes a participant identifier254and a talk time256that indicates how many minutes of the call250the participant252spoke. The participant database244includes a plurality of participants260. Each participant260includes a participant identifier254, a title264of the participant260, the subject matter expertise266of the participant260, as well as other information268related to the participant260, such as a geographic location, and/or an office or branch location of the participant.

The OS238generally manages various computer resources (e.g., network resources, file processors, and/or the like). The operating system238is configured to execute operations on one or more hardware and/or software modules, such as Network Interface Cards (NICs), hard disks, virtualization layers, firewalls and/or the like. Examples of the operating system238may include, but are not limited to, LINUX, MAC OSX, BSD, UNIX, MICROSOFT WINDOWS,10S, ANDROID and the like.

The networks206comprise one or more communication systems that connect computers by wire, cable, fiber optic and/or wireless link facilitated by various types of well-known network elements, such as hubs, switches, routers, and the like. The networks206may include an Internet Protocol (IP) network110, a public switched telephone network (PSTN)130, or other mobile communication networks listed above, and may employ various well-known protocols to communicate information amongst the network resources.

Before a conference call begins, the call history analyzer240predicts which participant274is likely to be the dominant speaker based on the total talk time256on previous calls250between the participants274of the current conference call. Based on the total talk time256from all calls250that include the participants274, a calculation is performed to determine the percentage of time in the previous conference each participant spoke. For example, it may be determined that over the past communications between Participant A, Participant B, and Participant C that Participant A spoke fifty (50) percent of the time, Participant B spoke thirty (30) percent of the time, and Participant C spoke twenty (20) percent of the time. In some embodiments, the call history analyzer240then weighs these percentages based on who is designated the host270of the current call, the subject matter expertise266of the participant based on the topic272of the current call, the title264of the participant based on the topic272, or other information, such as a geographic location, and/or an office or branch location of the participant, and the like. For example, if the topic272of the current call is product definition meeting, and Participant B is the Senior Project Director, Participant B may be weighted such that in the present meeting Participant B would be predicted to be the dominant speaker even though Participant B is typically second in total talk time. The information regarding which participant274is predicted to be the dominant speak, the second most dominant, and so on is sent to each user device202.

The information regarding the dominant speaker is transmitted based on the connection type of each user device202. For example, if the user device202is connected through the Internet, the information may be sent using a Transmission control Protocol/Internet Protocol (TCP/IP) connection.

The conference display module222of each user device202displays the participants on the display212in a hierarchy that indicates who the predicted dominant speaker is. In some embodiments, the dominant speaker is displayed having the largest percentage of area taken in the display device. In some embodiments, the dominant speaker is displayed having the uppermost position of the area in the display212in comparison to the remaining participants. In some embodiments, the dominant speaker is displayed having a graphically depicted border along a perimeter of the image of the dominant speaker.

The current call monitor246monitors which participants274are speaking during the call and their talk time276. If during talk time monitoring, the talk time of the predicted dominant speaker falls below a predefined threshold of his or her historically longest talk time, the hierarchical layout is adjusted. At such time as talk time of the predicted dominant speaker falls below the predefined threshold, the current participant with a talk time that is longer than the historically longest talk time of the predicted dominant speaker is displayed more prominently than the remaining participants. The predefined threshold may be from 60-80% of the historically longest talk time of the predicted dominant speaker. The hierarchical layout is adjusted more than once every x minutes, for example no more than every two minutes.

In some embodiments, the top y places in the hierarchical layout are never adjusted. For example, if a conference call includes fifteen (15) participants and the top eight (8) participants are displayed, the top, for example five (5) participants are fixed on the display. If a non-displayed subordinate speaker increases his or her talk time above a pre-defined threshold, the non-displayed subordinate speaker may be displayed in one of the lower, non-fix positions.

In some embodiments, the system is connected to a speech-to-text engine, for example Alexa®, Siri®, or a service such as Vonage® AI that interprets the quality of what each speaker is saying. If, based on technical or business keywords detected in the speech-to-text string, general vocabulary, and the like, a subordinate speaker may be deemed to have a higher quality speech. In such embodiment, hierarchical layout of the participants is adjusted with the subordinate speaker with the higher quality speech displayed more prominently. When the current call is ended, the current call monitor246stores information about the call in the call database242, including at least the talk time276for each participant274.

FIG. 3depicts a flow diagram of a method300for using predictive analysis to generate a hierarchical layout of participants in a video conference on display devices, in accordance with one or more embodiments of the present invention. The method300may be performed using one or more of the modules described above with respect toFIG. 2. The method300starts at step302and proceeds to step304.

At step304, a request is received to set up a video conference call. The video conference call is among three or more participants. The participants may have previously been involved in voice and/or video calls prior to the present call. For example, the call may be a weekly status meeting.

At step306, a dominant speaker of the three or more participants is predicted. In some embodiments, the participants are ranked based on who has the historically longest talk times during prior communications among the three or more participants, with the participant having the longest talk times being predicted as the dominant speaker. In some embodiments, the participants are ranked; based on one or more of call organizer, call topic, subject matter expertise as it relates to the call topic, and title of each participant, a geographic location, and/or an office or branch location of the participant. In some embodiments the participants are ranked based on who has the historically longest talk times during prior communications among the three or more participants, and then the rankings are weighted based on one or more of call organizer, call topic, subject matter expertise as it relates to the call topic, title of each participant, a geographic location, and/or an office or branch location of the participant.

At step308, the participants are displayed in a hierarchical layout on display devices of the participants with the predicted dominant speaker displayed more prominently than the remaining three or more participants. In some embodiments, the predicted prominent speak may be displayed by taking the largest percentage of area in the display device. In some embodiments, the predicted prominent speak may be displayed in the uppermost position of the area in the display device in comparison to the remaining participants. In some embodiments, the predicted prominent speak may be displayed with a graphically depicted border along a perimeter of the image of the dominant speaker.

At step310, the talk times of each of the three or more participants are monitored.

At step312, the hierarchical layout of the participants is adjusted if the talk time of the predicted dominant speaker is below a threshold amount, for example, 60%-80% of talk time of a subordinate speaker. For example, if the dominant speaker is speaking only 70% of his or her historically longest time, the display is adjusted to prominently display the current participant with a talk time that is longer than the historically longest talk time of the predicted dominant speaker.

At step314, it is determined whether the conference call has ended. If it is determined that the conference call is ended, then the method300proceeds to step316, where the conference information including the participants and total talk time for each participant is stored in a database to be used in predicting future dominant speakers. However, if at step314, it is determined that the conference call has not ended, then the method proceeds to step310and iterates until the conference call has ended.

The method300ends at step318.

FIG. 4is an exemplary diagram of a computer system400for using predictive analysis to generate a hierarchical layout of participants in a video conference on display devices in accordance to one or more embodiments of the present invention. The computer system400includes substantially similar structure comprising servers or electronic devices in the aforementioned embodiments.

Various embodiments of methods and system using predictive analysis to generate a hierarchical layout of participants in a video conference on display devices, as described herein, may be executed on one or more computer systems, which may interact with various other devices. One such computer system is computer system400illustrated byFIG. 4, which may in various embodiments implement any of the elements or functionality illustrated inFIGS. 1-3. In various embodiments, computer system400may be configured to implement methods described above. The computer system400may be used to implement any other system, device, element, functionality or method of the above-described embodiments. In the illustrated embodiments, computer system400may be configured to implement method300as processor-executable executable program instructions422(e.g., program instructions executable by processor(s)410) in various embodiments.

In the illustrated embodiment, computer system400includes one or more processors410a-410ncoupled to a system memory420via an input/output (I/O) interface430. Computer system400further includes a network interface440coupled to I/O interface430, and one or more input/output devices460, such as cursor control device460, keyboard470, and display(s)480. In some embodiments, the keyboard470may be a touchscreen input device.

In various embodiments, any of the components may be utilized by the system to authenticate a user as described above. In various embodiments, a user interface may be generated and displayed on display480. In some cases, it is contemplated that embodiments may be implemented using a single instance of computer system400, while in other embodiments multiple such systems, or multiple nodes making up computer system400, may be configured to host different portions or instances of various embodiments. For example, in one embodiment some elements may be implemented via one or more nodes of computer system400that are distinct from those nodes implementing other elements. In another example, multiple nodes may implement computer system400in a distributed manner.

In different embodiments, computer system400may be any of various types of devices, including, but not limited to, personal computer systems, mainframe computer systems, handheld computers, workstations, network computers, application servers, storage devices, a peripheral device such as a switch, modem, router, or in general any type of computing or electronic device.

In various embodiments, computer system400may be a uniprocessor system including one processor410, or a multiprocessor system including several processors410(e.g., two, four, eight, or another suitable number). Processors410may be any suitable processor capable of executing instructions. For example, in various embodiments processors410may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs). In multiprocessor systems, each of processors410may commonly, but not necessarily, implement the same ISA.

System memory420may be configured to store program instructions422and/or data432accessible by processor410. In various embodiments, system memory420may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing any of the elements of the embodiments described above may be stored within system memory420. In other embodiments, program instructions and/or data may be received, sent or stored upon different types of computer-accessible media or on similar media separate from system memory420or computer system400.

In one embodiment, I/O interface430may be configured to coordinate I/O traffic between processor410, system memory420, and any peripheral devices in the device, including network interface440or other peripheral interfaces, such as input/output devices450. In some embodiments, I/O interface430may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory420) into a format suitable for use by another component (e.g., processor410). In some embodiments, I/O interface430may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface430may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface430, such as an interface to system memory420, may be incorporated directly into processor410.

Input/output devices450may, in some embodiments, include one or more display devices, keyboards, keypads, cameras, touchpads, touchscreens, scanning devices, voice or optical recognition devices, or any other devices suitable for entering or accessing data by one or more computer systems400. Multiple input/output devices450may be present in computer system400or may be distributed on various nodes of computer system400. In some embodiments, similar input/output devices may be separate from computer system400and may interact with one or more nodes of computer system400through a wired or wireless connection, such as over network interface440.

In some embodiments, the illustrated computer system may implement any of the methods described above, such as the methods illustrated by the flowchart ofFIG. 3. In other embodiments, different elements and data may be included.