SELECTIVE CONTROL OF AUDIO IN VIRTUAL CONFERENCE ROOMS

Computer methods and systems allow for selective control of the audio and visual output of users in a virtual conference room. For example, users may be grouped into groups, and users within a group may see and hear other users within that group more prominently than users outside the group. Groups may be associated with other groups as peripheral, and users in a group associated as peripheral may see and hear other users in the peripheral group less prominently. Additionally, a user may be designated as a main speaker, and all users may see and hear that main speaker more prominently than other users in the virtual conference room. Further, a user may direct audio input to one or more users through a whisper. AI may be used to improve the system.

INTRODUCTION

Video conferencing has increasingly become an integral part of doing business. With the introduction of high-definition audio, users can now experience crystal-clear sound quality. In addition, most audio-conferencing systems have features such as call recording, virtual backgrounds, and mute options. Many audio-conferencing systems also integrate with other applications, such as instant messaging, file sharing, and screen sharing, making collaboration and communication more efficient.

One issue with the current state of video conferencing, however, is that participants who are in the same virtual “room” hear and see everyone equally, regardless of their “virtual” proximity or level of involvement in the conversation. This differs from the real-life experience of being in a physical room, where certain people's voices and appearances are more prominent while others are at the periphery.

The problems that arise in video conferencing can be frustrating and distracting for participants, ultimately leading to negative impacts on the overall flow of the conversation. For example, when participants talk over each other, it can be difficult for other participants to follow the conversation and make sense of what is being discussed. This can lead to misunderstandings, confusion, and missed opportunities to contribute to the conversation.

Similarly, when participants cannot hear each other clearly, it can be challenging to maintain engagement and focus. Participants may become distracted or disengaged, leading to a decrease in productivity and collaboration. Additionally, participants who are not heard clearly may feel left out of the conversation or undervalued, which can negatively impact morale and motivation.

In a virtual conference environment, using network bandwidth and computing processing power to render unwanted videos and broadcast unwanted audio further exacerbates this problem. For example, slower network connectivity and clunky computer experiences may result.

These issues are particularly important in contexts where effective communication is crucial, such as in business meetings, classrooms, or healthcare settings. As a result, addressing the challenges of video conferencing is an ongoing priority for the industry, and there is an ongoing need for innovation and technological advancements to ensure that virtual communication remains effective and efficient. It is with respect to these and other considerations that the technologies described below have been developed. Also, although relatively specific problems have been discussed, it should be understood that the examples provided are not meant to be limited to solving the specific problems identified in the introduction or elsewhere.

SUMMARY

Aspects of the technology include a computer-implemented method. The method includes receiving a plurality of requests to join a virtual meeting. The method also includes allowing access to the virtual meeting to at least a first user, a second user, a third user, and a fourth user based in part on in part the plurality of requests. The method also includes grouping the first user's audio input and the second user's audio input into a first audio input cluster. The method also includes grouping the third user's audio input and the fourth user's audio input into a second audio input group. The method also includes altering a first-user audio output to the first user such that the first audio group is louder than the second audio input group.

In aspects, the method also includes outputting instructions to the first user to display a visual representation of a second user more prominently than a visual representation of the third user or a visual representation of a fourth user. The method may also include receiving, from a whispering user, a request to private message a target user of a virtual conference call. The method may also include sending a request to accept the private message to a client application associated with the target user, receiving an indication of acceptance, and, based on receiving the indication of acceptance, setting a whispering environment to facilitate a private voice conversation between the target user and the whispering user. Setting the whispering environment may alert other members a group associated with the target user that the target user is in a private conversation. Alerting may include changing a video image of the target user to a still image. The method may also include associating a first user's input with the first user, analyzing, using a Deep Neural Network, the first user's input to determine the first user's topic of conversation, and suggesting a different group to the first user based on the determination. The method may also include receiving an indication that the first user wishes to change groups based on the suggesting operation.

Additionally/alternatively, aspects of the technology include a computer-implemented method. The method includes receiving, by a server, a request from a plurality of clients to join a virtual conference call. In some aspects, the plurality of clients includes a first client having a first input communication stream including audio and video data captured by the first client and a second client having a second input communication stream including audio and video data captured by the second client. The method also includes sending at least a portion of the audio and video data captured by the first client and at least a portion of the audio and video data captured by the second client to at least a portion of the other of the plurality of clients. The method also includes receiving a request from the first client to send a private whisper to the second client. The method also includes setting a whisper environment based on the request.

Setting the whisper environment may include reducing the amount of data of the audio and video data captured by the first client that is sent to the at least a portion of the other of the plurality of clients. Setting the whisper environment may also include reducing the amount of data of the audio and video data captured by the second client that is sent to the at least a portion of the other of the plurality of clients. The method may also include sending an indication to the at least a portion of plurality of other clients that the first client and the second client are in a whisper environment. The indication may be selected from the group consisting of: graphical indication, changing the video feed of the first client and the second client to a still image, and an audio indication. The method may also include before setting the whisper environment, sending an approval request to the second client and receiving, from the second client, an approval.

The computer-implemented methods may be stored on a computer-readable storage device that stores instructions that, when executed, perform the method.

SELECTIVE CONTROL OF AUDIO IN VIRTUAL CONFERENCE ROOMS

Aspects of the technology relate to video conferencing. In aspects, the technology provides a server the ability to selectively choose which members of a virtual conference room are focused, which are in the peripheral, and which are not displayed. In aspects of the technology, one or more servers adjusts the audio and visual prominence of participants in the conference room to highlight certain participants while reducing the volume and visual prominence of others. In examples, a user chooses a group to join. In examples, the members of that group are displayed more prominently to other ingroup members. Other members of the virtual conference may be displayed/heard less prominently (e.g., members or users who are in the peripheral groups to the group the user is in). Some participants of the conference call may not be displayed at all to a particular user.

For some applications, this technology allows users to have more control over their virtual communication environment, making it easier to follow conversations and stay engaged with other participants. For example, users can choose to focus on the speaker or speakers who are most relevant to the topic being discussed while minimizing distractions from other participants who may be less involved in the conversation of interest to the user.

One possible application of this technology is in educational settings, where instructors can use this feature to ensure that all students are able to hear and engage with the material being presented. By selectively highlighting certain students and reducing the volume of others, instructors can help minimize distractions and ensure that everyone is able to stay focused on the topic at hand. For example, the teacher may group students into groups, which will allow the students to hear other students within the group more prominently than others in the virtual classroom. The other group of students, however, may still be heard by outside-clustered students, though less prominently, thus replicating the experience of a classroom.

Overall, this technology represents a significant step forward in the field of video conferencing, providing users with more control over their virtual communication environment and helping to improve the overall effectiveness and efficiency of virtual communication. Moreover, using the technology, the server can save bandwidth by sending only information sufficient to display/broadcast users who are prominently displayed or displayed in the peripheral, and not necessarily all members of the conference call.

In particular, aspects of the technology relate to a computer method that may be used to selectively control audio and visual displays in a virtual conference room. The technology includes a method that groups members of a video conference into groups based on user selection. For example, a user may interact with a GUI to join a group of other members of the conference room. In other examples, a user with administrative capabilities (e.g., a teacher) may group other members into a group. In some examples, the user may opt to leave that group and join another group.

In additional aspects of the technology, a user may invite another member of the virtual conference room to the user's current group (or other group). Additionally, one user may whisper to another user. A whisper, in examples, is a directed audio and/or video (real-time or not) message to another user. The message may not be heard by certain members (e.g., one or more members in a group or all other members of the virtual conference call).

In examples, once the participants have been grouped, the method reduces the volume and visual appearance of all other members/other groups of the virtual conference room who are not a part of the group. In some applications, this helps to minimize distractions and ensure that participants can focus on the relevant parts of the conversation without being overwhelmed by extraneous audio and visual stimuli. Additionally, this may help in limiting network usage and computer processing usage by limiting the information sent to user devices (e.g., participant computers running client applications to facilitate the virtual conference call).

In additional examples, a computer method that can be used to selectively control audio in a virtual conference room involves giving priority to a main speaker by one or more users to be more prominent than all other members of the conference room. This method may, in examples, create an experience of a main speaker(s) being on stage and the other members being in a crowd, with the main speaker being the focus of attention. Once the main speaker(s) has been identified (e.g., through a user interface), the method increases the main speaker's audio volume and visual appearance, in examples, while decreasing the volume and visual appearance of all other members of the conference room (for each user, for example).

This approach helps to ensure that the main speaker is heard clearly and that their message is conveyed effectively, while still allowing other participants to be heard and seen to a lesser extent. It also helps to create a more natural and dynamic conversation flow, similar to what one might experience in a physical meeting room.

The technology can be customized to suit the specific needs of different users and contexts and can be implemented using a variety of software tools and platforms. For example, it can be used in business meetings or educational settings to ensure that the main speaker is able to deliver their message effectively or in large virtual events such as webinars or conferences where there is a need to prioritize certain speakers.

Overall, this technology represents an innovative and effective approach to selectively controlling audio in virtual conference rooms, helping to improve the quality and efficiency of virtual communication. For some applications, these improvements come along with the added benefit of reducing network bandwidth usage and computer processing resources for both server and participant computers.

These and various other features as well as advantages that characterize the systems and methods described herein, will be apparent from a reading of the following detailed description and a review of the associated drawings. Additional features are set forth in the description, which follows and, in part, will be apparent from the description or may be learned by practice with the technology. The benefits and features of the technology will be realized and attained by the structure, particularly pointed out in the written description and claims hereof, as well as the appended drawings.

It is to be understood that both the foregoing introduction and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the innovative technologies as claimed and should not be taken as limiting.

FIG.1illustrates a networked-computing environment100for facilitating a virtual conference call in which various technologies described herein may be employed. As illustrated,FIG.1includes a first participant computing device102, a second participant computing device104, a third participant computing device106, a fourth participant computing device108, a plurality of other participant computing devices110, a server112, and a database114, each of which is communicatively coupled to each other via a network126.

Participant computing devices may be any suitable type of computing device. For example, the computing device may be a desktop computer, a laptop, a computer, a tablet, a mobile telephone, a smartphone, a wearable computing device, or the like. The first participant computing device102is illustrated as a smartphone, the second participant computing device104is illustrated as a desktop computer, the third participant computing device106is illustrated as a laptop computer, and the fourth participant computing device108is illustrated as a tablet. The plurality of other particpant computing devices110may be any computer device. It will be appreciated that more or less computing devices may be present in a networked environment without deviating from the scope of the innovative technologies described herein.

In examples, participant computing devices have one or more executable programs or applications capable of interacting with one or more servers, such as server112, to allow a user of a participant computing device to participate in a video conference. For example,FIG.1illustrates a first client application122that is running on the fourth participant computing device108. The first client application122may be a downloadable application that is configured to run on the operating system of the fourth participant computing device108and allow a user to participate in a conference call. One skilled in the art will appreciate that other means of participating in the virtual conference call are contemplated, such as the use of a browser124operating on a first participant computing device102.

Participating with the virtual conference call is facilitated by, in examples, one or more servers. For example, one or more servers, such as server112, handles media relaying and processing. As an example, the server112may receive various audio and video streams from the participant devices, such as the first participant computing device102, the second participant computing device104, the third participant computing device106, the fourth participant computing device108, and a plurality of other participant computing devices110. The server112may then send various output streams of video/audio to the computing devices to cause the audio and/or video of certain conference call attendees to be more prominent as further described herein. In examples, the server112may handle audio and video streams using a variety of techniques, including mixing.

As illustrated, one or more servers, such as a server112, may perform a variety of other functions related to the conference call session. These functions include, but are not limited to, managing call initiation and termination for various participants, managing user authentication, stabilizing connections (e.g., such as by managing latency, jitter, and packet loss), terminating the call session, providing real-time transcription, noise suppression, and/or echo cancellation, synchronizing shared content such as screen sharing, presentations, and/or collaborative documents, maintaining the order of messages in a chat or instant messaging, recording and storage of conference call, and/or security and encryption. The server112may also perform other functions such as billing and usage reporting, tracking call metrics, and providing API integration capabilities for third-party applications, CRMs, calendaring, or other enterprise software.

A networked database114is illustrated. In aspects, the networked database114stores information such that the information is accessible over the network126by various devices, including the first participant computing device102, the second participant computing device104, a third participant computing device106, the fourth participant computing device108, a plurality of other participant computing devices110, a server112, be it through a local area network (LAN) or the internet, or other suitable network connection.

FIG.2is an example system200to provide selective control of audio and video streams in a virtual environment. As illustrated, the server208stores various engines in server memory216, including a whisper engine226, a group assignment engine228, a media input engine230, a media output engine232, an AI engine234, and an AI training engine236. The server memory216is in electronic communication with processor224. While illustrated as a single computer with a processor224, it will be appreciated that multiple servers and/or multiple processors may be implemented without deviating from the scope of the innovative technologies described herein.

In examples, the group assignment engine228assigns the clients running on each participant computer to one or more groups during a virtual conference call. In an example, a user interacts (through a touch screen or other input device) with the client application of a participant computer to select or otherwise indicate that the user wishes to be assigned to a group comprising other users in the virtual conference call. In examples, the group assignment engine228receives that input and assigns the participant computer to the group indicated by the user interaction. In some examples, a user may be defaulted to no group or a predetermined group. Group assignments may be used to determine the prominence of audio/video displayed/broadcast of other user(s) in the conference call.

In some examples, a group assignment engine228assigns users to groups and handles change requests as follows. The group assignment engine228may receive input from a client application, such as first client application250, second client application252, or third client application254, indicating that the user of the client application wishes to join a group of the virtual conference. The group assignment engine228then, in an example, associates that user with the group.

Group assignment engine228may also assign peripheral groups to other groups. One scheme for determining peripheral groups vis-a-vis other groups is a two-peripheral linear scheme. Such linear scheme may work as follows: when a first group is formed, that first group has no peripheral groups. When a second group is formed, the second group is peripheral to the first group and the first group is peripheral to the second group. When a third group is formed, that third group is peripheral to the first and second group. The second group will now be peripheral to the third group and the first group. The first group will be peripheral to the second and fist group. Thereafter, if another group is formed, that group will be added to the end, the previously last group will sever its peripheral connections to the first group and instead connect with the newly joined group, and the newly formed group will associate the first group as peripheral. Thus, the topography of a 6-group conference call, may look like:

where nodes that are connected by lines indicate a group being associated as peripheral to the connected group(s). If a new group, group 7 is formed, then the topography may then change to:

In the example illustrated, the server208is in electronic communication with a first participant computer202, a second participant computer204, and a third participant computer206. In this example, the conferencing server receives input from the first participant computer202via a first input communications channel238, receives input from a second participant computer204from a second input communications channel242, and receives input from a third participant computer206via a third input communications channel246.

Input communications channels, such as a first input communications channel238, includes information transmitted from participant computers. For example, audio information and visual information may be captured at various participant computers (e.g., via a microphone and/or camera in electronic communication with the participant computers), processed, and sent via the communications channel (through, for example, a network, such as the Internet) to the server208.

In an aspect of the technology, the media input engine230receives the various input communications channels and processes the input. For example, the audio and video input received from the various participant computers may be identified and associated with users and groups by the media input engine230.

In examples, a whisper engine226handles private messages from one user to another user in a virtual meeting. For example, a user may interact with a client application, such as a first client application250via a GUI, and cause a private message to be sent to another user, such as a second user on a second participant computer204interacting with a second client application252. In examples, the whisper engine226receives the indication that the first user wishes to whisper to the second user and handles the request. For example, the whisper engine may direct some or all information contained in first input communication channel238(e.g., audio and video content) to be directed only to the second participant computer204via the second output communication channel244. The whisper engine226may also send information to other client applications that are not a part of the private communication, such as third client application254operating on the third participant computer206. This information may be an indication that the two users are in private communication.

A media output engine232handles sending the appropriate output to various users in a virtual call setting based on group affiliation. In an example, the media output engine232may cause audio output and/or video output to be delivered and adjusted to users based on that user's association with a group. For example, a client application associated with a user may receive information to cause virtual call participants of that same group to be more prominently displayed/broadcast than other participants of a conference call. As another example, a client application may broadcast audio/video input of users who are in periphery groups less prominently than the audio/video of others in a group of users. Audio/video of users in groups that are not associated with a user group may not be sent to the user at all.

Also illustrated is an AI engine234and an AI training engine236. In aspects of the technology, the AI engine234analyzes the natural language of the group and performs numerous functions based on the analysis. For example, the AI engine234may change the heading of the group name to match the topic of conversation and may suggest users to join another group. The AI engine234may also suggest users form a sub or different group. Each of these suggestions is, in examples, based on an analysis performed by the AI engine234of the words exchanged in the virtual conference room by the various users. In examples, the AI training engine236may adapt the AI engine234by monitoring whether the users positively/negatively react to the changes/suggestions of AI engine234. For example, where the users click accept or do not revert the group name to another and/or previous group name, the AI training engine236may register that as a positive, tag the content used to generate the suggestion, and use that information to train the AI engine234.

Groups may also be associated with one another, such as a group being in the periphery of another group. Following this example, when a first group associates with a second group as peripheral, a user in the first group may be able to interact with users of the second group in a different manner than other users not in the peripheral use. In an example, the first group members may be able to see the second group members on a smaller portion of their screen, whisper to the second group members, join the second group, etc. A more fulsome explanation of group interactions is provided with reference to the figures below.

In the example illustrated, the first participant computer202has a first memory210operating a first client application250using a processor218, the second participant computer204has a second memory212operating a second client application252using a processor220, and a third participant computer206has a third memory214operating a third client application254using a processor222. It will be appreciated that client applications may be run using multiple processors across distributed systems as further described herein.

FIGS.3A and3Billustrate examples of handling a change in group assignment of users over time that may be employed using the technologies described herein.FIGS.4A and4Bare tables that correspond to the examples described inFIGS.3A and3B.FIG.3Aillustrates five groups of users at a time, T1. These groups include a first group348, a second group350, a third group352, a fourth group354, and a plurality of other groups358. It will be appreciated that more or less groups may be present during a virtual conference call without deviating from the scope of the innovative technologies described herein.

As illustrated inFIG.3A, first group348includes user A326, a user B332, a user C328, and a user D330at time T1. T1 corresponds to the table illustrated inFIGS.4A and4B. As shown in column three406A labeled “group,” and column five410A, each of the users within a group receive priority audio and video of the other users in the group. For example, an inspection of column one402A and column five410A illustrates user A having priority audio/video from B, C, and D, user B having priority audio/video of users A, C, and D, user C having priority audio/video of users A, B, and D, and user D having priority audio/video of users A, B, and C. Communication to each user is facilitated by the server302using the communication streams indicated by observing the column402A and the communications stream indicated in column two404A. For example, communications stream304is used to allow user B to communicate to server302, which is also illustrated inFIGS.3A and3B.

Column four408A indicates the groups which are associated as proximate to the group indicated in column three406A. For example, user K is associated with first group348and is proximate to groups344and354. Indeed,FIG.4Aillustrates that, user K will receive priority audio/video content from users I and J at time T1 and will receive peripheral audio/video content from users A, B, C, D.

As illustrated, T1 is a time in which a user H340has yet to be assigned a group. In the example provided, the user H340does not have any priority content from any user because user H340is not in a group. In other examples, a host or designated user is the automatic priority audio/video content for any new users. Also as illustrated, the user H has no peripheral audio/visual content from any users. In some examples, user H will be assigned peripheral groups as discussed above or may be assigned peripheral content by a predetermined list. Alternatively, the user may be prompted to select, through interaction with a GUI at a client application, one or more groups to add as peripheral before choosing a group to join. This will, in examples, allow the user to receive peripheral content from that group.

Additionally illustrated are a plurality of other groups358which may make up one or more users356in one or more groups. The plurality of other groups358is illustrated as not being designated as being peripheral to any of the first group348, the second group350, the third group352, or the fourth group354. Thus, it is contemplated that all groups may necessarily be successfully linked by peripheral groups, though in some examples, they are (e.g., in the linear scheme described above).

FIG.3Billustrates an example state of a grouping during a virtual conference call at time T2. As illustrated user H340is now in the fourth group354. In this example, user H340now receives priority audio/video content based on user H340's association with fourth group354. H340now receives priority audio/video content from users I, J, and K, as illustrated. Additionally, user H340also receives peripheral audio/video content from users A, B, C, and D because user H340is now a member of fourth group354. As illustrated (though it need not be), any user who is a member of a group with fourth group354designated as proximate will now receive peripheral audio/video content from user H340. In this example, that include users A, B, C, and D.

FIG.5is an example method500for assigning a user of a virtual call to a group. Method500begins with accept user into video conference operation502. In operation502, a client application, such as the client applications described above, connects with a server (e.g., the one or more servers described herein). In an example, an application from a participant computer sends a request to the server through a client application. The request contains, in examples, details such as the user's identity and the conference room the user wishes to join (as transmitted through the application, for example). A particular protocol may be used, such as WebRTC. The server then processes this request and establishes the appropriate connections sufficient to let the user in the conference room.

Method500then proceeds to associate user to a group operation504. In operation504, the user is associated with a group. As described further herein, a user's association with a group may be used to determine, at least in part, the prominence and/availability of audio/video of other members of the virtual call. For example, a conference call application may display ingroup users in the middle of the screen and at a louder volume than other members of the conference call. Assignment of a user to a group may occur by default. For example, a user may be assigned to a group with or by a conference participant who is the administrator. Alternatively, the user may enter the conference call as a group of 1 (only the particular user). The user may then, through interaction with a GUI, join another group. Alternatively, the AI engine may assign a user a group based on a natural language analysis of previous audio/chats used by the user in other conference calls (or other gathered data).

Method500then proceeds to send priority information operation506. Priority information may be sent to those users in the same group as a particular user. In operation506, the client application receives priority audio/video information. In an example, the priority audio/video information is sufficient for the application to display one or more other users of the conference more prominently than other users of the conference call and/or more loudly than other members of the conference call.

Method500then proceeds to determination508, where it is determined whether other users are in a group associated as peripheral to the original user's group. In an example, a group may be associated with other groups as peripheral. This association may be preset by an administrator of the program, or the AI Engine may automatically update peripheral group information based on a natural language analysis of the communications occurring in each group (and the related nature of the conversation). In other aspects, a linear peripheral group scheme may be employed as described herein. If groups are identified as peripheral, and users are in those peripheral groups, then method500proceeds to send peripheral user information operation510.

In operation510, information sufficient for a user to display other users as peripheral are sent. This may include instructions to display audio/video of peripheral users (e.g., users that are in groups peripheral to a first user). For example, peripheral users may be displayed in smaller windows, without video content (e.g., displaying only photographs or avatar of users), and with a softer audio.

After operation510, or if determination operation508is no, then the method500proceeds to receive additional group selection determination512. If additional group selection is received (e.g., through a GUI at a participant device), then method500returns to associate user with group operation504. If not, the method500ends.

FIG.6is an example method600of organizing input communication streams for users of a virtual call. Method600begins with receive incoming communications stream from one or more users operation602. In examples, communication streams (such as audio, video, and textual input captured by a participant computer) are encoded and transmitted to one or more servers.

Method600then proceeds to associate operation604, where each of those incoming audio/video streams of a user is associated with a group. In examples, for each communication stream received from a user in a virtual conference call, the server may associate that stream with the user and/or with a group. Such association may occur by using one or more of relationships, keys, and references.

FIG.7is an example method700for determining priority content to deliver to a particular user of a group. Method700begins with identify ingroup user operation702. In operation702, other members who are in the same group as the particular user are identified, for example, by a server. It will be appreciated that “particular user” has no meaning other than to note that the particular user is different from other users. Other users in the group may be individually identified as well, and may be referred to as a second user, a third user, etc., (with no ordered meaning implied), a different user, another particular user, and the like. In aspects of the technology, one or more servers may identify each member who is in the same group as the particular user.

Method700then proceeds to send priority output operation704. In operation704, audio/video output of each other user in the group is sent to the particular user. Priority audio/video output may be output sufficient to cause the audio/video of the other members of the group to be displayed/broadcast more prominently than other users who are not in the group.

Method700then proceeds to determination706. In determination706, it is determined whether there are additional users in the group. If so, the next user is identified and method700then returns to operation704, before which the additional identified user is set to the particular user. If not, the method ends.

FIG.8is an example method800for determining peripheral content to deliver to users of a particular group. Method800begins with identify peripheral groups operation802. In operation802, one or more groups are identified as peripheral to the particular group. This may be identified by accessing a database storing data indicating which groups are associated as peripheral. It will be appreciated that particular group has no meaning other than to note that the particular group is different from other groups. Other groups in the virtual conference call may be individually identified as well, and may be referred to as a second group, a third group, etc., (with no ordered meaning implied), a different group, another particular group, and the like.

Method800then proceeds to identify peripheral call participants operation804. In operation804, participants who are members of the one or more peripheral groups identified in operation802are identified to form peripheral participants. This may occur by cataloging, tagging, or otherwise recording the current participants of the one or more peripheral groups.

Method800then proceeds to send operation806. In operation806, one or more servers sends each user of the particular group information sufficient to display peripheral content. This may be information sufficient for users of the particular group to display video (or images) and/or broadcast the audio of each of the peripheral participants. For example, one or more servers may have received input communication streams from each of the peripheral participants as described herein. The server may then use those input communication streams to send each user of the particular group peripheral content information of the peripheral participants. The method then ends.

FIG.9is a method for sending a private message to another user of a virtual call. Method900begins with receive indication to send private user a message operation902. This indication may be sent by a user (the whispering user) interacting with a client application on a participant computing device. For example, the whispering user may click on an image or video of another participant (the target participant) in the virtual conference call to indicate that the whispering user wishes to send a private message to the target user.

Method900then optionally proceeds to accept determination904. In determination904, communication may be sent to the target user's client application indicating that the whispering user wishes to send a message to the target user. In aspects of the technology, the target user's application may wait for an indication of acceptance from the target user. This may occur by, for example, the target user clicking accept or otherwise interacting with a GUI and/or the client application.

After receive indication902, or in the event that the accept determination was accepted in determination904, operation then proceeds to set whisper environment906. In operation906a whisper environment is set. This may be set by the server sending control information to the application of the whispering user and the target user so that they can only hear each other. In examples, users who are in the peripheral and/or ingroup users may be sent an indication noting that the target user and the whispering user are in a private conversation. In an example, each in group user display an icon indicating that the target user and/or the whispering user are in a private conversation. In some aspects where video of the users is displayed, that video may cease to be delivered to other members of the group and or members of peripheral groups of the target user and/or whispering user. In examples, an image may be sent instead of a video. This may both prevent distraction and decrease network usage. The method then ends.

FIG.10is a method for analyzing a virtual call using artificial intelligence. Method1000begins with receive communication input operation1002. In operation1002, communications may be received from one or more users of a virtual call. Communications may be received as part of input communication streams, as further provided herein. In aspects, the communications are audio inputs from users, which may be converted to text or another recognizable form suitable for processing by an AI engine, such as the AI engine described herein. Additionally, chat from the virtual call may also be received.

Method1000then proceeds to associate operation1004. In operation1004, one or more users who generated the content (e.g., by talking in a virtual conference call or typing in chat) is associated with the content. Additionally, other information, such as group name, peripheral groups, and other ingroup users and peripheral users to the content generating user may be associated with each user, the group, and/or peripheral groups. This information forms at least a part of user content information.

Method1000then proceeds to analyze user content information operation1006. In operation1006, the content is analyzed to determine one or more topics of conversation in the various groups of a virtual conference call. For example, the DNN may identify the topic of conversation of a particular user and/or a particular group. In aspects of the technology, a Deep Neural Network (“DNN”) is used. For example, a DNN might be trained on large datasets of user content information, where each text is labeled with its corresponding one or more topics. As it learns, the network hones its ability to recognize patterns and structures in the text that indicate a particular topic for a user and/or group. When presented with new, unseen text, such as new user content information, the trained DNN then analyzes the language and outputs the most likely topic of conversation based on the patterns it has previously learned.

Method1000then proceeds to take action operation1008. In operation1008, an action may be taken based on topics identified in operation1006. For example, a group in a virtual call may have a name indicating the topic. That name may be different from the topic identified in operation1008. In such a case, the action may be to change the group name to the topic identified in operation1008. Additionally/alternatively, the DNN may have identified that a user is discussing one topic, whereas the rest of the group is discussing another. In such a case, a prompt may be sent to the user to indicate other groups that are discussing the same topic as the user. After operation1008, the method ends.

FIG.11is a method1100for training a DNN using feedback from a virtual call. The DNN may be the same as described above. Method1100begins with take action operation1102. Take action operation1102may be the same or similar as the take action operation1008discussed above.

Method1100then proceeds to capture feedback operation1104. For example, where the topic of a group was changed to match the identified topic, feedback may include a user manually changing the topic back to the previous or different topic. In some instances, not receiving a change for a certain period of time, such as 5 minutes, 10 minutes, etc. may also be observed. Additionally, where a different group was suggested to a user based on a user topic, it may be captured where the user changes group or does not for a set period of time.

Method1100then proceeds to tag data operation1106. When the captured data indicates that the DNN performed adequately (e.g., when the user switches group or the group name is not changed within a certain period of time), the user content information and suggestion is sent to a DNN to update the model as tagged data.

FIG.12Ais an example diagram of a distributed computing system1200A in which aspects of the present innovative technology, including the virtual call environment described above, may be implemented. According to examples, any of computing devices, such as a modem1202A, a laptop computer1202B, a tablet1202C, a personal computer1202D, a smart phone1202E, and a server1202F, may contain engines, components, etc. for instantiating the virtual call environment described above. Additionally, according to aspects discussed herein, any of the computing devices described or referred to herein may contain necessary hardware for implementing aspects of the disclosure. Any and/or all of these functions may be performed, by way of example, a network of servers and/or one server when computing devices request or receive data from external data provider by way of a network1220.

Turning toFIG.12B, one embodiment of the architecture of a system for performing the technology discussed herein is presented. Virtual call environments interacted with, requested, and/or edited in association with one or more computing devices may be stored in different communication channels or other storage types. For example, data may be stored using a directory service, a web portal, a mailbox service, an instant messaging store, or a compiled networking service for facilitating a virtual call environment as described herein. The distributed computing system1200B may be used for running the various engines to instantiate the virtual call environment, such as the engines described with reference toFIG.2above. The computing devices1218A,1218B, and/or1218C may provide a request to a cloud/network1215, which is then processed by a network server1220in communication with an external data provider1217. By way of example, a participant computing device may be implemented as any of the systems described herein and embodied in the personal computing device1218A, the tablet computing device1218B, and/or the mobile computing device1218C (e.g., a smart phone). Any of these aspects of the systems described herein may obtain content from the external data provider1217.

In various examples, the types of networks used for communication between the computing devices that make up the present invention include, but are not limited to, an Internet, an intranet, wide area networks (WAN), local area networks (LAN), virtual private networks (VPN), GPS devices, SONAR devices, cellular networks, and additional satellite based data providers such as the Iridium satellite constellation which provides voice and data coverage to satellite phones, pagers, and integrated transceivers, etc. According to aspects of the present disclosure, the networks may include an enterprise network and a network through which a client computing device may access an enterprise network. According to additional aspects, a client network is a separate network accessing an enterprise network through externally available entry points, such as a gateway, a remote access protocol, or a public or private Internet address.

Additionally, the logical operations may be implemented as algorithms in software, firmware, analog/digital circuitry, and/or any combination thereof, without deviating from the scope of the present disclosure. The software, firmware, or similar sequence of computer instructions may be encoded and stored upon a computer readable storage medium. The software, firmware, or similar sequence of computer instructions may also be encoded within a carrier-wave signal for transmission between computing devices.

Operating environment1300typically includes at least some form of computer-readable media. Computer-readable media can be any available media that can be accessed by a processor such as processing device1380depicted inFIG.13and processor1402shown inFIG.14or other devices comprising the operating environment1300. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program engines or other data. Computer storage media includes, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information. Computer storage media does not include communication media.

FIG.14illustrates one aspect of a computing device1400, which may be used to implement aspects of the present disclosure, including any of the plurality of computing devices described herein with reference to the various figures and their corresponding descriptions. The computing device1400illustrated inFIG.14can be used to execute an operating system1396, application programs1398, and program modules1303(including the engines described with reference toFIG.2) described herein.

The computing device1310includes, in some embodiments, at least one processing device1380, such as a central processing unit (CPU). A variety of processing devices are available from a variety of manufacturers, for example, Intel, Advanced Micro Devices, and/or ARM microprocessors. In this example, the computing device1310also includes a system memory1382, and a system bus1384that couples various system components including the system memory1382to the at least one processing device1380. The system bus1384is one of any number of types of bus structures including a memory bus, or memory controller; a peripheral bus; and a local bus using any of a variety of bus architectures.

Examples of devices suitable for the computing device1310include a server computer, a pod, a mobile-monitoring device, a mobile deployment device, a static-monitoring device, a desktop computer, a laptop computer, a tablet computer, a mobile computing device (such as a smart phone, an iPod® or iPad® mobile digital device, or other mobile devices), or other devices configured to process digital instructions.

Although the exemplary environment described herein employs a hard disk drive as a secondary storage device, other types of computer readable storage media are used in other aspects according to the disclosure. Examples of these other types of computer readable storage media include magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, compact disc read only memories, digital versatile disk read only memories, random access memories, or read only memories. Additional aspects may include non-transitory media. Additionally, such computer readable storage media can include local storage or cloud-based storage.

A number of program engines can be stored in the secondary storage device1392or the memory1382, including an operating system1396, one or more application programs1398, other program modules1303(such as the software engines described herein), and program data1302. The computing device1310can utilize any suitable operating system, such as Linux, Microsoft Windows™, Google Chrome™, Apple OS, and any other operating system suitable for a computing device.

According to examples, a user provides inputs to the computing device1310through one or more input devices1304. Examples of input devices1304include a keyboard1306, a mouse1308, a microphone1309, and a touch sensor1312(such as a touchpad or touch sensitive display). Additional examples may include input devices other than those specified by the keyboard1306, the mouse1308, the microphone1309and the touch sensor1312. The input devices are often connected to the processing device1380through an input/output (I/O) interface1314that is coupled to the system bus1384. These input devices1304can be connected by any number of I/O interfaces1314, such as a parallel port, serial port, game port, or a universal serial bus. Wireless communication between input devices1304and the interface1314is possible as well, and includes infrared, BLUETOOTH® wireless technology, cellular and other radio frequency communication systems in some possible aspects.

In an exemplary aspect, a display device1316, such as a monitor, liquid crystal display device, projector, or touch-sensitive display device, is also connected to the computing system1300via an interface, such as a video adapter1318. In addition to the display device1316, the computing device can include various other peripheral devices, such as speakers or a printer.

When used in a local area networking environment or a wide area networking environment (such as the Internet), the computing device1310is typically connected to a network such as network1220shown inFIGS.12A and12Bthrough a network interface, such as an Ethernet interface. Other possible embodiments use other communication devices. For example, certain aspects of the computing device1310may include a modem for communicating across the network. The computing device1310typically includes at least some form of computer-readable media. Computer-readable media includes any available media that can be accessed by the computing device1310. By way of example, computer-readable media include computer-readable storage media and computer-readable communication media.

The computing device1310illustrated inFIG.13is also an example of programmable electronics, which may include one or more such computing devices, and when multiple computing devices are included, such computing devices can be coupled together with a suitable data communication network so as to collectively perform the various functions, methods, or operations disclosed herein.

FIG.14is a block diagram illustrating additional physical components (e.g., hardware) of a computing device1400with which certain aspects of the disclosure may be practiced. Computing device1400may perform these functions alone or in combination with a distributed computing network such as those described with regard toFIGS.12A and12Bwhich may be in operative contact with personal computing device1218A, tablet computing device1218B and/or mobile computing device1218C which may communicate and process one or more of the program engines described herein.

In a basic configuration, the computing device1400may include at least one processor1402and a system memory1410. Depending on the configuration and type of computing device, the system memory1410may comprise, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory1410may include an operating system1412and one or more program modules1414. The operating system1412, for example, may be suitable for controlling the operation of the computing device1400. Furthermore, aspects of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and are not limited to any particular application or system.

The computing device1400may have additional features or functionality. For example, the computing device1400may also include additional data storage device (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG.14by storage1404. It will be well understood by those of skill in the art that storage may also occur via the distributed computing networks described inFIG.12AandFIG.12B. For example, computing device1400may communicate via network1220inFIG.12Aand data may be stored within network servers1206and transmitted back to computing device1400via network1220if it is determined that such stored data is necessary to execute one or more functions described herein. Additionally, computing device1400may communicate via network1220in and data may be stored within network server1206and transmitted back to computing devices1202A-1202F via network1220if it is determined that such stored data is necessary to execute one or more functions described herein.

As stated above, a number of program engines and data files may be stored in the system memory1410. While executing the at least one processor1402, the program modules1414may perform processes including, but not limited to, the aspects described herein.

One skilled in the art will appreciate the foregoing detailed description is provided by way of illustration and not limitation. The examples presented herein are intended to facilitate a clear understanding of the innovative technologies disclosed, and they are not exhaustive of the potential embodiments or examples encompassed by the scope of this disclosure. Those skilled in the art will readily recognize alternative implementations and variations that remain within the broad principles of the invention. Therefore, it should be understood that the scope of the present disclosure encompasses all such modifications and alternative embodiments as fall within the true spirit and scope of the appended claims.