Natural language markup for meetings

A natural language markup for meetings is introduced that facilitates planning and facilitation of online meetings. Shared content is obtained during an online meeting. The shared content is shared by a first participant in the online meeting for display on devices of one or more second participants in the online meeting. A visual object is detected in the shared content and additional content is obtained based on detecting the visual object. The additional content is transmitted with the shared content for display on the devices of the one or more second participants.

TECHNICAL FIELD

The present disclosure relates to online video meetings/conferences.

BACKGROUND

When presenting shared content during an online meeting, multiple participants in the online meeting may be responsible for presenting different aspects of the presentation or shared content. Changes to the shared content may occur last minute and it may be difficult to synchronize versions of the shared content. In addition, different participants may prepare content to present using different applications. The technical meeting setups may become the focus of the meeting, potentially derailing the meeting from its intended purpose.

DETAILED DESCRIPTION

Overview

In one embodiment, a method is provided for improved planning and facilitation of online meetings by introducing a natural language markup for meetings. The method includes obtaining shared content during an online meeting, the shared content being shared by a first participant in the online meeting for display on devices of one or more second participants in the online meeting; detecting a visual object in the shared content; obtaining additional content based on detecting the visual object; and transmitting the additional content with the shared content for display on the devices of the one or more second participants.

Example Embodiments

Online meetings may be used to present material by sharing content from one or more participants' user devices. In some cases, different participants in the online meeting may be responsible for presenting different aspects of the material during the online meeting. Currently, online meeting services/applications do not offer smart actions based on presentation material during online meetings.

Often in team meetings, project updates, company all-hands meetings, and other type of meetings, multiple people in a team may be responsible for presenting different parts of a presentation or discussion. Changes may occur last minute, and it may be hard to synchronize content among different presenters. Tools exist that provide multi-user updates to a document, but these tools only work if all users are sharing content using the same format or application (e.g., PowerPoint). Some contributors may present using slides, other contributors may present using a word processing document, PDF or web page, and some contributors may want to use a whiteboard for their portion of the presentation. Without scripting the online meeting, participants must manually pass the share between themselves. In other words, a first participant shares content for their portion of the presentation and when it is a second participant's turn to present, the first participant stops sharing content and the second participant starts sharing content. The technical meeting setup can become a focus of the meeting, potentially derailing the meeting from its intended purpose.

When different participants are presenting different parts of a presentation, the presenter may be expected to be highlighted, for example by having the presenter's video stream presented on the screen or stage. The highlighted participants may or may not be expected to speak. Currently it is not possible to plan the meeting so that when a certain point is reached in the presentation, a person or a selected group of people will be brought forward or highlighted. In addition, several useful sharing scenarios, such as natively displaying a web page inline in a videoconference or simultaneous sharing of multiple content sources, are unavailable. Additionally, a facilitator or host of a meeting is currently unable to set up an online meeting so that a participant is prompted to share material at a planned point.

In many cases, videoconferences hosted by major enterprises are unaware of the meetings' agendas beyond basic details, such as start time and duration. Therefore, online meeting services/applications do not guide participants through the meetings (e.g. by keeping time during particular segments). As such, the online meeting services/applications fall short of meetings that are hosted by professional, human meeting facilitators.

Embodiments described herein provide for improved planning and facilitation of online meetings by being contextually aware of the content and the participants by introducing a natural language markup for meetings. The markup language consists of text and visual objects/artifacts present in shared content that define actions to be performed during the online meeting. The markup is introduced to presentation material of any file format, and is interpreted by a meeting server or client/device (e.g., using artificial intelligence (AI) processes) in advance or in real time during a meeting.

In particular, embodiments described herein provide for automatically detecting a visual object in shared content and performing one or more actions based on detecting the visual object. An endpoint device and/or a meeting server may include a machine learning (ML) module that analyzes content being shared by a participant of an online meeting and detects predefined visual objects using AI technology. When a visual object is detected, the ML module replaces an area containing the visual object (or a different area on the screen) with second content. The second content may be, for example, live video feed of a participant in the online meeting, a document stored in a cloud service or at a user device, an application, etc. According to some embodiments, when the visual object is detected, the ML module may identify additional information in the content to identify the second content to replace the area containing the visual object. According to other embodiments, the second content is identified based on type of the visual object.

According to embodiments described herein, the natural language markup consists of text and visual artifacts/objects that define actions to be performed in-meeting by an online meeting service. A user device or meeting server interprets the text and visual artifacts/objects in any shared content, regardless of a file format or application type of the shared content. The user device or meeting server performs actions as specified by the markup to facilitate meetings. The actions may include, for example, replacing the visual artifacts/objects with video feed(s) or other shared content (in a same or different file format or application type as the shared content), presenting video feed(s) or other shared content next to the shared content, providing instructions or reminders to participants in the online meetings, dividing content among multiple screens, or performing additional actions.

Reference is first made toFIG.1.FIG.1shows a block diagram of a system100that is configured to provide online meeting services using a natural markup language. The system100includes one or more meeting server(s)110and a plurality of endpoint devices120-1to120-N (sometimes referred to herein individually as endpoint device120or collectively as endpoint devices120) that communicate with meeting server(s)110via one or more networks130. The meeting server(s)110are configured to provide an online meeting service for hosting a communication session among endpoint devices120-1to120-N. The meeting server(s)110includes a machine learning (ML) module160for identifying and interpreting visual objects and/or text in content shared during an online meeting and performing actions based on the visual objects and/or text. In some embodiments, one or more of the endpoint devices120-1to120-N may include a ML module160, as illustrated by the dashed lines around the ML module160in endpoint device120-1. Although in the example described with respect toFIG.1, the functions of ML module160are performed at meeting server(s)110, the functions of ML module160may be performed at an endpoint device120-1to120-N or a different device not illustrated inFIG.1.

Each of endpoint devices120-1to120-N may be a tablet, laptop computer, desktop computer, Smartphone, virtual desktop client, virtual whiteboard, or any user device now known or hereinafter developed. Endpoint devices120-1to120-N may have a dedicated physical keyboard or touch-screen capabilities to provide a virtual on-screen keyboard to enter text. Endpoint devices120-1to120-N may also have short-range wireless system connectivity (such as Bluetooth™ wireless system capability, ultrasound communication capability, etc.) to enable local wireless connectivity with a video endpoint device in a meeting room or with other user devices in the same meeting room. Endpoint devices120-1to120-N may store content (e.g., a presentation, a document, images, etc.) for sharing the content with other user devices during an online meeting or communication session. In some embodiments, endpoint devices120-1to120-N may be endpoint devices for facilitating communication with meeting server(s)110during the online meeting. Endpoint devices120-1to120-N may additionally connect to user devices (not illustrated inFIG.1) for sharing content stored on the user devices during the online meeting.

In some cases, endpoint devices120-1to120-N may be videoconference endpoints designed for personal use (e.g., a desk device used by a single user) or for use by multiple users (e.g., a videoconference endpoint in a meeting room). In some embodiments, endpoint devices120-1to120-N may be configured to open content to display or share (e.g., when a digital whiteboard is accessed directly on endpoint device120).

In the example illustrated inFIG.1, a user (e.g., a meeting coordinator, host, or co-host) of endpoint device120-1may be presenting content and, at140, endpoint device120-1may transmit shared content with visual objects to meeting server(s)110via network(s)130. Meeting server(s)110may transmit the shared content to endpoint devices120-2to120-N for display on endpoint devices120-2to120-N. ML module160may detect the visual objects in the shared content. For example, ML module160may analyze the shared content using AI technology that is trained to recognize the visual objects. The visual objects may include graphics and/or text.

ML module160may interpret the graphics and/or text and perform actions based on information associated with the graphic and/or text. For example, a particular graphic may indicate that meeting server(s)110is to perform a particular action. As another example, the text may describe the actions that are to be performed. In some cases, a visual object may include a graphic and text. In this example, ML module160may detect the graphic and, based on detecting the graphic, identify the text and an action to take based on instructions in the text.

In some embodiments, the action may include transmitting instructions, reminders, or other information to one or more participants in the online meeting. In other embodiments, the action may include replacing a visual object with other content or transmitting the other content for display next to or on top of the shared content. For example, the graphic and/or text may indicate that a visual object in the shared content shared by the user of endpoint device120-1is to be replaced with other content, such as content obtained from another endpoint device (e.g., endpoint device120-2). As another example, the graphic and/or text may indicate that other content is to be placed next to or on top of a portion of the shared content.

As shown at150, meeting server(s)110obtains the content from endpoint device120-2. Meeting server(s)110replaces the visual object with the content obtained from endpoint device120-2or places the content obtained from endpoint device120-2next to or on top of the shared content from endpoint device120-1. The content obtained from endpoint device120-2may be, for example, a video feed of a user of endpoint device120-2, shared content (e.g., slides, a word processing document, etc.), content from a webpage, content from another application, or another type of content. In some embodiments, the text may direct the meeting server(s)110to instruct a user of endpoint device120-2to share content and the content shared by the user of endpoint device120-2may displayed with the content shared from endpoint device120-1(e.g., the content shared by the user of endpoint device120-2may replace the visual objects in the content shared by the user of endpoint device120-1).

Meeting server(s)110performs the action, such as replacing the visual object with the content or displaying the content with the shared content from endpoint device120-1, and transmits the content shared by endpoint device120-1with the content from endpoint device120-2to endpoint devices120-2to120-N. Endpoint devices120-2to120-N display the shared content from endpoint device120-1with the content from endpoint device120-2on the same screen. In some embodiments, endpoint devices120-2to120-N display the shared content from endpoint device120-1with the content from endpoint device120-2replacing the visual objects in the shared content from endpoint device120-1. In this scenario, the users of endpoint devices120-2to120-N will not see the visual objects (i.e., the graphics and/or text) and, instead, will see the content from endpoint device120-2that replaces the visual objects.

By automatically performing the actions based on identifying and interpreting the visual objects, meetings in which more than one participant is presenting content may run more smoothly. Instead of performing a first share with a first participant and then performing a second share with a second participant, the second participant may seamlessly share content within the content shared by the first participant. In addition, a video feed of one or more presenters may automatically be brought forward when it is time for the one or more presenters to present content. In this way, the focus of the meeting may be on the content presented instead of the mechanics of how the content will be presented.

Reference is now made toFIG.2.FIG.2illustrates a user interface200depicting an example in which visual objects are replaced with video feeds of users in response to detecting graphics and/or text in content shared by a host of an online meeting.

In the example illustrated inFIG.2, a coordinator or host of an online meeting may share content including shared content202. Shared content202includes images220-1,220-2, and220-3that correspond to participants participating in the online meeting and an avatar icon220-4that includes a name of a user (e.g., Participant 9) who is participating in the online meeting. Images220-1to220-3may be, for example, photographic images of the participants. Shared content202additionally includes text210. In this example, the text reads “Enable Inline.” The inclusion of the predefined command in text210(e.g., “Enable Inline”) triggers an AI interpretation of the shared content. In this example, ML module160(which is located at meeting server(s)110, endpoint devices120, etc.) may identify the text210that reads “Enable Inline” as a predefined command to overlay the images220-1to220-3and the avatar icons220-4with live video feeds of the corresponding participants. In other examples, different text may correspond to different actions.

In the example illustrated inFIG.2, ML module160may use, for example, facial recognition technology to match an image220-1to220-3with a video feed of a corresponding participant. In some embodiments, ML module160may match an image220-1to220-3or an avatar icon220-4with a corresponding video feed based on a name associated with the image220-1to220-3or the avatar icon220-4. For example, ML module160may identify the video feed corresponding to the avatar icon220-4by identifying the video feed that corresponds to “Participant 9.” ML module may replace the images220-1to220-3and the avatar icon220-4with the corresponding video feeds. For example, image220-3may be replaced with video feed230-1.

By replacing the images and/or avatar icons with the video feeds of the corresponding participants in the online meeting, participants presenting particular content may be brought forward to present the content. In addition, when new shared content is presented, the participants are automatically removed from the shared content. In this way, particular users may be easily highlighted for presenting certain content.

Reference is now made toFIG.3.FIG.3illustrates a user interface300depicting an example in which a visual object is replaced with shared content in response to detecting graphics and/or text in content shared by a meeting host.

In the example illustrated inFIG.3, a coordinator or a host of an online meeting shares content that includes an icon310, such as a mountain icon, and text320. Although a mountain icon is depicted inFIG.3, any icon may be used at a placeholder for shared content. In addition, in some embodiments, different icons may represent different actions to perform. In this example, the ML module160may identify the icon310as a placeholder for a shared content. In other words, in this example, the mountain icon may be a predefined icon indicating that shared content is to be obtained from a participant in the online meeting and used to replace the area around the mountain icon. In some embodiments, in response to detecting the mountain icon, ML module160may identify text in the content shared by the meeting host or coordinator (i.e., the main shared content) to determine where or how to obtain the shared content to replace the mountain icon.

In response to detecting the icon310, ML module160may identify and parse the text320. In the example illustrated inFIG.2, text320says “Participant 1, please unmute & share https://jira.com/our-scrum-board.” By parsing text320, ML module160may transmit a message to a participant in the online meeting (e.g., Participant 1) requesting that the participant who is next to share content unmute his or her microphone and share content (e.g., a slide, a web page, a word processing document, etc.). In this example, the participant is requested to share a URL and, if the participant acknowledges or selects an option to accept the instructions, the participant's default web browser may automatically open for navigating to a webpage associated with the URL. Alternatively, the participant may open a web browser and navigate to the webpage. When the webpage is open, the participant may share the content of the webpage with the other participants in the meeting.

When the participant shares the contents of the webpage, the contents may be displayed in place of icon310. In this example, the area inside dotted line330may be replaced with the contents of the web page. The content shared by the participant (e.g., Participant 1) is subordinate to the content shared by the coordinator or the host. In other words, Participant 1 ‘ s sharing ends when the coordinator or the host moves forward in the main presentation. For example, if the coordinator or host moves to a new slide in a presentation, Participant 1’ s content is no longer shared in the online meeting.

Although the example illustrated inFIG.3shows content from a webpage being shared within the host's main share during the online meeting, content from different applications may be shared within the host's main share. Additionally, according to embodiments described herein, the subordinate shared content may be from a different application than the coordinator or host's shared content. For example, the host may be presenting the main share using slides from a first application and the participant (e.g., Participant 1) may share subordinate content using a different application (e.g., a word processing application, a spreadsheet application, a whiteboard application, a web browser application, etc.). In this way, information that may be best presented using a particular application may easily be shared even when the host is sharing using a different application.

Reference is now made toFIG.4.FIG.4illustrates a user interface400depicting an example in which content from multiple participants replaces visual objects in content shared by a meeting host or coordinator.

In the example illustrated inFIG.4, a meeting host or coordinator shares content that includes icons310-1,310-2,310-3, and310-4and text410-1,410-2,410-3, and410-4. In this example, the ML module160may identify the icons310-1to310-4as placeholders for shared content. In the example illustrated inFIG.4, icons310-1to310-4are mountain icons, but any icons may be included in the shared content. As described above with respect toFIG.3, different icons may represent different actions.

In some embodiments, ML module160may transmit instructions to the participants to share the content. For example, similar to the example described above with respect toFIG.3, ML module160may transmit a message to Participant 1 with instructions to share the content. Participant 1 may open and share the content and Participant 1's shared content may replace the area within dotted line420-1. Similarly, content shared by Participant 4 may replace the area inside dotted line420-2, content shared by Participant 2 may replace the area within dotted line420-3, and content shared by Participant 3 may replace the area within dotted line420-4. In some embodiments, ML module160may not transmit a message to the participants with instructions to share the content. Instead, the participants may share the content without receiving the instructions and ML module160may replace icons310-1to310-4with the appropriate shared content. In some embodiments, an image or live video feed of the participant sharing the content may be shown alongside the shared content.

In some embodiments, the content shared by the different participants may be from different applications. In addition, the content shared by the different participants may be from different applications than the main content. For example, the meeting host or coordinator may share the main content using a slideshow application and the subordinate shared content may be content from a word processing application, a spreadsheet application, a web browsing application, a whiteboard application, and/or different applications.

By sharing content from multiple people at the same time, different data or aspects associated with a presentation may be shared at the same time. By using the natural markup language, the content may be easily and quickly be shared by multiple participants using one or more applications at the same time.

Reference is now made toFIG.5.FIG.5illustrates a user interface500depicting an example in which content is divided between screens in response to detecting graphics and/or text in content shared by a meeting host or coordinator.

In the example illustrated inFIG.5, a meeting host or coordinator shares content that includes icons510-1and510-2and content520-1and520-2. Content520-1includes icon310and text320. In this example, the ML module160may identify the icons510-1and510-2and determine, based on the icons510-1and510-2, that content520-1and520-2are to be divided across multiple screens or display areas. As shown inFIG.5, icon510-1reads “Screen1content below” and icon510-2reads “Screen2content below.” Therefore, ML module160identifies icons510-1and510-2and, based on parsing the text in the icons, displays content520-1on a first screen or display area and content520-2on a second screen or display area of participants' devices120-2to120-N.

In addition, ML module160may identify icon310and text320in content520-2. Based on the icon310and the text320, ML module160may request that Participant 1 share the content in the website identified in text320in a manner described above inFIG.3. In this example, participants (i.e., participants with multiple screens) participating in the online meeting may see content520-1on a first screen and the website shared by Participant 1 on a second screen. AlthoughFIG.5illustrates an example in which one of the screens displays content shared from a participant in the online meeting, in other embodiments, content520-1and520-2may be any type of content. For example, content520-1and520-2may both be content shared by the meeting host, one of content520-1or520-2may be content from an application (e.g., a whiteboard application), or different types of content may be displayed on different screens or display areas.

In some embodiments, when a participant does not have multiple screens, network server(s)110may present content520-1and520-2on a single screen. In this scenario, content520-1may be displayed in a first content area on the screen and content520-2may be displayed in a second content area on the screen. In some embodiments, content520-1may fill a first screen or the first content area if a participant has granular layout controls. Similarly, content520-2may fill a second screen or the second content area if the participant has granular layout controls.

Reference is now made toFIG.6.FIG.6illustrates a user interface600depicting an example in which an application is launched and content from the application is shared in response to detecting graphics and/or text in content shared by a meeting host or coordinator. User interface600includes shared content and video feeds630-1,630-2,630-3,630-4, . . .630-N of participants in the online meeting.

In the example illustrated inFIG.6, a meeting host or coordinator may share content that includes graphic610and text620. In this example, graphic610corresponds to a whiteboard application. In some embodiments, different graphics may correspond to different applications. In addition, text620says “Start whiteboard.” ML module160may identify graphic610and/or text620and, based on the identification, automatically launch a whiteboard application and share content from the whiteboard application. In this example, since the text620does not identify a participant in the online meeting, meeting server(s)110may launch the whiteboard application on a user device associated with the meeting host or coordinator (e.g., endpoint device120-1) and share the content from the whiteboard application within dotted line640.

Since graphic610is associated with the whiteboard application, meeting server(s)110may launch whiteboard application on the meeting host's user device and share the content with other participants when user interface600does not include text620. In other embodiments, the whiteboard application may be launched and the content from the whiteboard application may be shared with other participants if a different graphic (such as, for example, the mountain graphic illustrated inFIG.3) that indicates that additional content is to replace the area surrounding the graphic is displayed and text620reads “Start whiteboard.” In other embodiments, if text620identifies a participant other than the host of the meeting, meeting server(s)110may instruct the identified participant to launch the whiteboard and the content from the whiteboard of the identified participant may replace the area within dotted line640.

Reference is now made toFIG.7.FIG.7illustrates a user interface700depicting an example in which a visual object is replaced with shared content in response to detecting graphics and/or text shared by a meeting host or coordinator.

In the example illustrated inFIG.7, a meeting host or coordinator shares content that includes icons310and text710. In this example, the ML module160may identify the icon310as a placeholder for shared content. In the example illustrated inFIG.7, icon310is a mountain icon, but any icons may be included in the shared content. As described above, different icons may represent different actions.

After identifying icon310, ML module160may identify and parse text710. As illustrated inFIG.7, text710says “Open locally: https://slid.do/#098-665-234.” Based on parsing the text710, ML module160may launch a web viewer on the local device (e.g., endpoint device120-1associated with a meeting host or coordinator) to navigate to the URL identified in text710. The content from the web site corresponding to the URL is displayed for all meeting participants in an area inside dotted line720.

Reference is now made toFIG.8.FIG.8illustrates a user interface800depicting an example in which a visual object is replaced with shared content in response to detecting graphics and/or text in content shared by a meeting host or coordinator.

In the example illustrated inFIG.8, a meeting host or coordinator shares content that includes icons310and text810. In this example, the ML module160may identify the icon310as a placeholder for shared content. In the example illustrated inFIG.8, icon310is a mountain icon, but any icons may be included in the shared content. As described above, different icons may represent different actions.

After identifying icon310, ML module160may identify and parse text810. As illustrated inFIG.8, text810says “Open locally: https://youtu.be/6ZrO90AI0c8.” Based on parsing the text810, ML module160launches a web viewer on the local device (e.g., endpoint device120-1associated with a meeting host or coordinator) to navigate the URL identified in text810. The specified media (e.g., a video) from the web site corresponding to the URL is displayed for all meeting participants in an area inside dotted line820. In this example, instead of displaying all of the content from the webpage corresponding to the URL for all participants in the meeting, ML module160recognizes that the web page corresponding to the URL is displaying media, such as a streaming video, and shares only the media for the participants in the online meeting.

Reference is now made toFIG.9.FIG.9illustrates a user interface900depicting an example in which video streams of participants replace a portion of a screen based on detecting a graphic or text in content shared by a meeting host or coordinator.

In the example illustrated inFIG.9, a meeting host or coordinator is sharing content that includes a quick response (QR) code910. ML module160may detect QR code910and may parse the QR code910to identify the content of the QR code910. The content of the QR code910may be plain text, XML text, etc. Particular keywords may be used to distinguish the QR code910from other QR codes. For example, an XML root node “<stageInfo>” may indicate that particular meeting participants are to be brought forward or to the “stage” for presenting information (such as information associated with the content being shared by the meeting host or coordinator). The content of the QR code910contains participant information (e.g., email addresses, employee identifier, etc.), sync stage information (whether to automatically synchronize to other participants), etc.

The QR code910includes the following exemplary information:

By parsing the information in QR code910, ML module160identifies that participants Participant 4 (with email address participant4@abc.com) and Participant 7 (with email address participant7@abc.com) are to be brought forward to present content and the stage information is to be synchronized to other participants. As shown inFIG.9, video feed920(e.g., Participant 4's video feed) and video feed930(e.g., Participant 7's video feed) are brought forward and placed on the screen next to the shared content. In this example, video feeds920and930replace the area to the right of the shared content, but video feeds920and930may replace other areas of user interface900. In some embodiments, video feeds920and930may replace graphics within the content shared by the meeting host or coordinator. QR code910may be blurred or cropped so other participants cannot view QR code910. In other embodiments, video feeds920and930may replace QR code910so the other participants may view video feeds920and930instead of QR code910.

In this example, when the meeting host or coordinator shares different content (e.g., by moving to a next slide in a presentation), the video feeds920and930will no longer be highlighted and the video feeds will once again be in the filmstrip of video feeds associated with the online meeting.

Reference is now made toFIG.10.FIG.10illustrates a user interface1000depicting an example in which video streams of participants replace a portion of a screen based on detecting a graphic or text.

In the example illustrated inFIG.10, a meeting host or coordinator is sharing content that includes a box1010including text1012. ML module160may detect box1010and parse text1012. For example, ML module160may use optical character recognition (OCR) or another technology to detect the text1012inside box1010. The text1012may include information associated with an action to perform. In some embodiments, different properties (e.g., color) of box1010may be associated with different actions (such as highlighting or bringing to the stage participants of the online meeting). The text1012may include, for example, participant information (e.g., email addresses, employee identifier, etc.), sync stage information (whether to automatically synchronize the stage to other participants), etc. In the example illustrated inFIG.9, text1012says “participant4@company.com,” “participant7@company.com,” and “AutoSync.” In this example, the text1012indicates that the video feeds of Participant 4 (corresponding to the email address participant4@company.com) and Participant 7 (corresponding to the email address participant7@company.com) are to be highlighted or brought to the stage and the stage is to be synchronized with other participants in the online meeting.

As shown inFIG.10, video feed1020(e.g., Participant 4's video feed) and video feed1030(e.g., Participant 7's video feed) are brought forward and placed on the screen next to the shared content. In this example, video feeds1020and1030replace the area to the right of the shared content, but, in other examples, video feeds1020and1030may replace other areas of user interface1000. In some embodiments, video feeds1020and1030may replace graphics within the content shared by the meeting host or coordinator. Box1010and/or text1012may be blurred or cropped so other participants cannot view box1010and/or text1012. In other embodiments, video feeds1020and1030may replace box1010and text1012so the other participants may view video feeds1020and1030instead of box1010and text1012.

In this example, when the meeting host or coordinator shares different content (e.g., by moving to a next slide in a presentation), the video feeds1020and1030will no longer be highlighted and the video feeds will once again be in the filmstrip of video feeds associated with the online meeting.

In some embodiments, in a meeting where too many participants are participating to show all participants in the filmstrip, text, such as text1012, may be used to identify the participants whose video feeds or images are to be prioritized or displayed in the filmstrip. For example, if the text1012in the shared content says “Team: Participant 4 & Participant 7,” ML module160may identify the video feeds or images of Participant 4 and Participant 7 and promote the video feeds so the video feeds become visible even when Participant 4 and Participant 7 are not speaking. Promoting the video feeds may entail including the video feeds in the visible part of the filmstrip, enlarging the video feeds, moving the video feeds, etc. ML module160may recognize the text1012even if the text1012is not within box1010. For example, if the text1012is in a predetermined format, ML module160may perform actions based on identifying the text without a graphic.

Reference is now made toFIG.11.FIG.11illustrates an exemplary agenda1100associated with an online meeting.

In an example associated withFIG.11, ML module160may detect and interpret agenda1100. Agenda1100includes items1102,1104,1106,1108,1110, and1112. Each item1102-1112includes a title associated with a portion of a meeting, a length of time associated with the portion of the meeting, and a presenter associated with the portion of the meeting. Agenda1100additionally indicates that the meeting facilitator (e.g., a meeting host or coordinator) is Participant 1. Based on agenda1100, ML module160may perform actions, such as sending messages to the meeting facilitator and/or presenter to keep the meeting running on time. For example, after Participant 4 has been presenting for 3 minutes of the allotted 5 minutes, ML module160, meeting server(s)110, or an endpoint device120may send a notification to Participant 4 with text, such as, “There are 2 minutes left of your allotted presentation time.” ML module160, meeting server(s)110, or an endpoint device120may additionally send a message to Participant 1 saying “There are 2 minutes left of Participant 1's allotted presentation time.” In this way, the presenter and/or meeting host may be kept aware of the remaining time in order to ensure the meeting stays on track and is running smoothly.

Reference is now made toFIG.12.FIG.12is a flow diagram illustrating a method1200of obtaining and displaying content based on detecting a visual object in shared content during an online meeting, according to an embodiment. Method1200may be performed by endpoint devices120-1to120-N and/or meeting server(s)110.

At1210, shared content is obtained during an online meeting. The shared content is shared by a first participant in the online meeting for display on devices of one or more second participants in the online meeting. For example, a host or organizer of an online meeting may share content using endpoint device120-1for display on endpoint devices120-2to120-N of other participants in the online meeting.

At1220, a visual object is detected in the shared content. For example, a ML module, such as ML module160, may analyze the shared content using an artificial intelligence process that is trained to recognize the visual object. The visual object may be, for example, a graphic, a QR code, text, or another visual object.

At1230, additional content may be obtained based on detecting the visual object. For example, the visual object may be associated with a particular type of content, the visual object may indicate the additional content and/or a user associated with the additional content, the ML module160may parse text to identify the additional content based on detecting the visual object, or the additional content may be obtained in additional ways. The additional content may be video feeds associated with participants in the online meeting, content shared by a participant in the online meeting, content from an application identified by the visual object, content from a web page, media obtained from a web page or another source, or a different type of content.

At1240, the additional content is transmitted with the shared content for display on the devices of the one or more second participants. For example, in one embodiment, the additional content may replace the visual object in the shared content and the shared content with the additional content may be shared for display on endpoint devices120-2to120-N. In another embodiment, the additional content may be placed next to the shared content for display on endpoint devices120-2to120-N.

The markup language described with respect toFIGS.1-12may enable a meeting facilitator to plan content sharing, plan participation, prioritize participants' video streams, control a meeting, populate a layout area, perform a simple survey, show specific, local xStatus, or perform additional actions. When planning content sharing, the facilitator may share predefined content (referenced by, for example, URL or application name), share content from predefined participants (reference by, for example, a name or email address), position shared content in a particular layout area, position multiple pieces of content simultaneously, share multiple pieces of content across multiple screens, open a whiteboard application for all participants, open a local web view for all participants, open a web application for all participants, open a media file for all participants (e.g., a piece of music that starts synchronous playback), present “contextually transparent” slides (e.g., minimize presentation) that make content sharing temporarily stop to focus on presenter/participants, present a “smaller presentation window” mode when simple content is show, but the focus is primarily on the presenter/participants, etc.

When planning participation, the facilitator may position specific participants in a particular layout area, position undefined participants in a particular layout area, mute and request unmuting of various audience groups, let participants in/out from the greenroom (backstage), call participant based on spoken or written words, alert participants about upcoming agenda items, inform participants about elapsed time in a current segment, etc. When prioritizing participants' video streams, the facilitator may infer which participants are to be prioritized based on content (e.g., based on whether the participants' names are on the slide).

When controlling a meeting, the facilitator may switch layout mode (e.g., People focus on/off), on an early slide, inform overall experience (e.g., “formal mode” shows name label as “J. Smith,” “informal mode shows name label as “Jane S.,” and “colleague mode” hides labels, “main participants” informs the names of the participants that should eb prioritized when possible), start, pause, resume, and stop recording (e.g., “Starting recording in 5 seconds, click here to cancel”), show chat, set camera presets (e.g., to view physical whiteboard, scene, audience, prototype, track a moving participant, etc.), present teleprompter text to planned speaker (e.g., text and name of speaker is in a dedicated area, which is masked or cropped away when viewed by other audience members), plan breakout rooms for participant segments, etc.

The facilitator may populate a layout area with a list of all current participants, a list of all current participants with contact information retrieved from respective local devices, a list of all participants and when they entered and left a meeting, calendar availability of all participants (e.g., this week or during the summer holiday), a geographical map with pinned locations for all participants, a local time for a particular participant (e.g., to be shown under the participant's feed), a summary slide that shows all action items, tasks, notes, and highlights submitted during the meeting, etc.

The facilitator may perform a simple survey with the results shown on the next/subsequent slide and stored in chats. The survey types may include a vote between options (result: vote count for each option), rate 1-5 stars (result: mean value), participants' individual comments (result: list of comments), participants' individual keywords (result: word cloud), etc. A facilitator may additionally show specific local xStatus (e.g., for debugging purposes; only shown locally, but potentially with “copy value” affordance).

In some embodiments, if the “markup area” (e.g., graphic, text, QR code, etc.) is not in an area that is to be replaced by other content, the markup area may be masked when presenting the slide by, e.g., filling in the slide with the general background color, blurring the markup area, etc. If the markup area is in a margin, the markup area may be cropped.

Referring toFIG.13,FIG.13illustrates a hardware block diagram of a computing/computer device1300that may perform functions of a video endpoint device or an end device associated with operations discussed herein in connection with the techniques depicted inFIGS.1-12. In various embodiments, a computing device, such as computing device1300or any combination of computing devices1300, may be configured as any devices as discussed for the techniques depicted in connection withFIGS.1-12in order to perform operations of the various techniques discussed herein.

In at least one embodiment, the computing device1300may include one or more processor(s)1302, one or more memory element(s)1304, storage1306, a bus1308, one or more network processor unit(s)1310interconnected with one or more network input/output (I/O) interface(s)1312, one or more I/O interface(s)1314, and control logic1320. In various embodiments, instructions associated with logic for computing device1300can overlap in any manner and are not limited to the specific allocation of instructions and/or operations described herein.

In at least one embodiment, processor(s)1302is/are at least one hardware processor configured to execute various tasks, operations and/or functions for computing device1300as described herein according to software and/or instructions configured for computing device1300. Processor(s)1302(e.g., a hardware processor) can execute any type of instructions associated with data to achieve the operations detailed herein. In one example, processor(s)1302can transform an element or an article (e.g., data, information) from one state or thing to another state or thing. Any of potential processing elements, microprocessors, digital signal processor, baseband signal processor, modem, PHY, controllers, systems, managers, logic, and/or machines described herein can be construed as being encompassed within the broad term ‘processor’.

In at least one embodiment, memory element(s)1304and/or storage1306is/are configured to store data, information, software, and/or instructions associated with computing device1300, and/or logic configured for memory element(s)1304and/or storage1306. For example, any logic described herein (e.g., control logic1320) can, in various embodiments, be stored for computing device1300using any combination of memory element(s)1304and/or storage1306. Note that in some embodiments, storage1306can be consolidated with memory element(s)1304(or vice versa), or can overlap/exist in any other suitable manner.

In at least one embodiment, bus1308can be configured as an interface that enables one or more elements of computing device1300to communicate in order to exchange information and/or data. Bus1308can be implemented with any architecture designed for passing control, data and/or information between processors, memory elements/storage, peripheral devices, and/or any other hardware and/or software components that may be configured for computing device1300. In at least one embodiment, bus1308may be implemented as a fast kernel-hosted interconnect, potentially using shared memory between processes (e.g., logic), which can enable efficient communication paths between the processes.

In various embodiments, network processor unit(s)1310may enable communication between computing device1300and other systems, entities, etc., via network I/O interface(s)1312(wired and/or wireless) to facilitate operations discussed for various embodiments described herein. Examples of wireless communication capabilities include short-range wireless communication (e.g., Bluetooth), wide area wireless communication (e.g., 4G, 5G, etc.). In various embodiments, network processor unit(s)1310can be configured as a combination of hardware and/or software, such as one or more Ethernet driver(s) and/or controller(s) or interface cards, Fibre Channel (e.g., optical) driver(s) and/or controller(s), wireless receivers/transmitters/transceivers, baseband processor(s)/modem(s), and/or other similar network interface driver(s) and/or controller(s) now known or hereafter developed to enable communications between computing device1300and other systems, entities, etc. to facilitate operations for various embodiments described herein. In various embodiments, network I/O interface(s)1312can be configured as one or more Ethernet port(s), Fibre Channel ports, any other I/O port(s), and/or antenna(s)/antenna array(s) now known or hereafter developed. Thus, the network processor unit(s)1310and/or network I/O interface(s)1312may include suitable interfaces for receiving, transmitting, and/or otherwise communicating data and/or information in a network environment.

I/O interface(s)1314allow for input and output of data and/or information with other entities that may be connected to computer device1300. For example, I/O interface(s)1314may provide a connection to external devices such as a keyboard1325, keypad, a touch screen, and/or any other suitable input and/or output device now known or hereafter developed. This may be the case, in particular, when the computer device1300serves as a user device described herein. In some instances, external devices can also include portable computer readable (non-transitory) storage media such as database systems, thumb drives, portable optical or magnetic disks, and memory cards. In still some instances, external devices can be a mechanism to display data to a user, such as, for example, a computer monitor, a display screen, such as display1330shown inFIG.13, particularly when the computer device1300serves as a user device as described herein. Display1330may have touch-screen display capabilities. Additional external devices may include a video camera1335and microphone/speaker combination1340. WhileFIG.13shows the display1330, video camera1335and microphone/speaker combination1340as being coupled via one of the I/O interfaces1314, it is to be understood that these components may instead be coupled to the bus1308.

In various embodiments, control logic1320can include instructions that, when executed, cause processor(s)1302to perform operations, which can include, but not be limited to, providing overall control operations of computing device; interacting with other entities, systems, etc. described herein; maintaining and/or interacting with stored data, information, parameters, etc. (e.g., memory element(s), storage, data structures, databases, tables, etc.); combinations thereof; and/or the like to facilitate various operations for embodiments described herein.

FIG.14illustrates a block diagram of a computing device1400that may perform the functions of the meeting server(s)110described herein. The computing device1400may include one or more processor(s)1402, one or more memory element(s)1404, storage1406, a bus1408, one or more network processor unit(s)1410interconnected with one or more network input/output (I/O) interface(s)1412, one or more I/O interface(s)1414, and meeting server logic1420. In various embodiments, instructions associated with the meeting server logic1420is configured to perform the meeting server operations described herein, including those depicted by the flow chart for method1200shown inFIG.12.

In one form, a computer-implemented method comprising: obtaining shared content during an online meeting, the shared content being shared by a first participant in the online meeting for display on devices of one or more second participants in the online meeting; detecting a visual object in the shared content; obtaining additional content based on detecting the visual object; and transmitting the additional content with the shared content for display on the devices of the one or more second participants.

In one example, the shared content is associated with a first application and the additional content is associated with a second application. In another example, obtaining the additional content comprises: detecting information associated with the additional content in the shared content; and obtaining the additional content based on the information. In another example, obtaining the additional content includes transmitting a request to a second participant of the one or more second participants to share the additional content. In another example, the visual object includes a photographic image of a second participant in the online meeting and the additional content includes a video feed of the second participant.

In another example, the additional content includes additional shared content, the additional shared content being shared by a second participant of the one or more second participants. In another example, the visual object is associated with a particular application and wherein obtaining the additional content includes launching the particular application to obtain the additional content. In another example, detecting the visual object comprises analyzing the shared content using an artificial intelligence process that is trained to recognize the visual object. In another example, the visual object includes a quick response (QR) code that includes information associated with the additional content.

In another example, the computer-implemented method further comprises replacing the visual object with the additional content in the shared content. In another example, the computer-implemented method further comprises: detecting, in the shared content, a second visual object associated with second content and a third visual object associated with third content, the second visual object indicating that the second content is to be displayed in a first area on the devices of the one or more second participants, and the third visual object indicating that the third content is to be displayed in a second area on the devices of the one or more second participants; and transmitting the second content for display in the first area and the third content for display in the second area on the devices of the one or more second participants.

In another form, an apparatus is provided comprising: a memory; a network interface configured to enable network communication; and a processor, wherein the processor is configured to perform operations comprising: obtaining shared content during an online meeting, the shared content being shared by a first participant in the online meeting for display on devices of one or more second participants in the online meeting; detecting a visual object in the shared content; obtaining additional content based on detecting the visual object; and transmitting the additional content with the shared content for display on the devices of the one or more second participants.

In yet another form, one or more non-transitory computer readable storage media are provided that are encoded with instructions that, when executed by a processor, cause the processor to execute a method comprising: obtaining shared content during an online meeting, the shared content being shared by a first participant in the online meeting for display on devices of one or more second participants in the online meeting; detecting a visual object in the shared content; obtaining additional content based on detecting the visual object; and transmitting the additional content with the shared content for display on the devices of the one or more second participants.

Variations and Implementations