AI-DRIVEN ADAPTIVE USER INTERFACE AMELIORATION IN SCREEN SHARING

According to one embodiment, a method, computer system, and computer program product for adaptive user interface amelioration during a screen sharing session is provided. The embodiment may include receiving shared content during a screen sharing session. The embodiment may also include identifying one or more device characteristics for a user device associated with a user and one or more user profile characteristics for the user. The embodiment may further include extracting topical elements within the received shared content. The embodiment may also include identifying a location of one or more objects and/or one or more text elements in the shared content. The embodiment may further include modifying a viewing area of the shared content on a display screen of the user device.

BACKGROUND

Web conferencing, including but not limited to audio conferencing and video conferencing, may relate to any communication between two or more individuals over a network, such as the Internet, where the participating members utilize audio and/or video streams. Web conferencing allows for a dynamic interface between users in separate locations to efficiently communicate as if they were in person by allowing face-to-face interactions and the sharing of files between participating users. Popular web conferencing applications include Zoom® (Zoom and all Zoom-based trademarks and logos are trademarks or registered trademarks of Zoom Video Communications Inc. and/or its affiliates), Webex® (Webex and all Webex-based trademarks and logos are trademarks or registered trademarks of Webex Communications, Inc. and/or its affiliates), GoToMeeting® (GoToMeeting and all GoToMeeting-based trademarks and logos are trademarks or registered trademarks of Citrix Online, LLC. and/or its affiliates), and FaceTime® (Facetime and all Facetime-based trademarks and logos are trademarks or registered trademarks of Apple Inc. and/or its affiliates) among others.

SUMMARY

According to one embodiment, a method, computer system, and computer program product for adaptive user interface amelioration during a screen sharing session is provided. The embodiment may include receiving shared content during a screen sharing session. The embodiment may also include identifying one or more device characteristics for a user device associated with a user and one or more user profile characteristics for the user. The embodiment may further include extracting topical elements within the received shared content. The embodiment may also include identifying a location of one or more objects and/or one or more text elements in the shared content. The embodiment may further include modifying a viewing area of the shared content on a display screen of the user device.

DETAILED DESCRIPTION

Embodiments of the present invention relate to the field of computing, and more particularly to web conferencing. The following described exemplary embodiments provide a system, method, and program product to, among other things, to dynamically adapt the layout and focus of the visible area of a display screen shown during a screen sharing session of a web conference customized for each user. Therefore, the present embodiment has the capacity to improve the technical field of web conferencing by providing a dynamic and customizable viewing experience that is specific to each user's needs and specific device capabilities.

As previously described, web conferencing, including but not limited to audio conferencing and video conferencing, may relate to any communication between two or more individuals over a network, such as the Internet, where the participating members utilize audio and/or video streams. Web conferencing allows for a dynamic interface between users in separate locations to efficiently communicate as if they were in person by allowing face-to-face interactions and the sharing of files between participating users. Popular web conferencing applications include Zoom®, Webex®, GoToMeeting®, and FaceTime® among others.

To provide a better user experience when consuming the same digital content across a wide variety of devices with varying form factors from laptops, tablets, and smartphones, application with responsive and adaptive user interfaces emerged. While the application itself was designed to be responsive, the content created by a user of the application is often not, and, therefore, might not offer the best user experience across different devices. For example, screen sharing of content via a web conference that allows users with devices of different form factors may be responsive to the form factor of the presenter or other web conference participants, but may result in a suboptimal user experience. As an additional example, a slide presentation shared via a web conferencing application may not provide an optimal user experience for a meeting participant who has joined the web conference via a smartphone or tablet when the screen sharing participant is using a laptop or desktop computer with different form factors (e.g., screen dimensions).

The user experience inefficiencies may be further amplified when the shared content contains decipherable information required for interpretation and/or understanding by the viewing participants. In such situations, participants affected by inefficiencies in the screen sharing session may need to zoom in to selective/important parts of the shared screen to effectively follow along with the sharing participant's presentation. This suboptimal user experience may lead to other issues such as user physical and/or mental fatigue after prolonged exposure. As such, it may be advantageous to, among other things, dynamically adapt the layout of content shared via a screen sharing session to meet the accessibility needs of multiple participants viewing the shared screen content using devices with different form factors.

According to at least one embodiment, an adaptive user interface amelioration program may be an edge computing application implementation or a part of a web conferencing client that takes in various inputs, such as, but not limited to, the current conversation/discussion between web conference participants, video content related to a screen sharing session (e.g., image frames), the type of end user device utilized by a web conference participant, and a user profile and other characteristics (e.g., distance of a user from a display screen, ocular strain or fatigue experienced by a user, user light sensitivity, and background lighting in the environment around the user). The adaptive user interface amelioration program may analyze and classify current speech transmitted during the web conference, classify objects from video stream images, map classifications of objects between the classified speech and images, and, depending on a type of device utilized by a user and user characteristics, determining what displayed objects need to be focused on or highlighted using a filtered list of matching objects. The adaptive user interface amelioration program may then modify the user interface of the web conferencing program to focus or zoom into a section of the screen determined to be topical to the current conversation between the web participants.

WAN 102 is any wide area network (for example, the internet) capable of communicating computer data over non-local distances by any technology for communicating computer data, now known or to be developed in the future. In some embodiments, the WAN 102 may be replaced and/or supplemented by local area networks (LANs) designed to communicate data between devices located in a local area, such as a Wi-Fi network. The WAN 102 and/or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and edge servers.

According to at least one embodiment, the adaptive user interface amelioration program 150 may receive media transmitted during a web conference screen sharing session and, using natural language processing (NLP)-driven text summarization, topical analysis, optical character recognition, NLP-drive image translation, and image object detection, identify content on a sharing participant's display screen that is the topic of the current conversation between the web conference participants. The adaptive user interface amelioration program 150, knowing the device constraints and formatting for the user device associated with one or more web conference participants, may modify (e.g., zoom and/or shift focus) the user device display screen for each web conference participant to optimize the presented screen share for the participant that focuses on the identified content.

Additionally, prior to initially performing any actions, the adaptive user interface amelioration program 150 may perform an opt-in procedure. The opt-in procedure may include a notification of the data the adaptive user interface amelioration program 150 may capture and the purpose for which that data may be utilized by the adaptive user interface amelioration program 150 during data gathering and operation. Furthermore, notwithstanding depiction in computer 101, the adaptive user interface amelioration program 150 may be stored in and/or executed by, individually or in any combination, end user device 103, remote server 104, public cloud 105, and private cloud 106. The adaptive user interface amelioration method is explained in more detail below with respect to FIGS. 2, 3A, and 3B.

Referring now to FIG. 2, an operational flowchart for an adaptive user interface amelioration process 200 according to at least one embodiment. During an initial setup phase to establish analyzation of language and images shared during a live screen sharing session, the adaptive user interface amelioration program 150 may leverage, configure, and/or integrate speech-to-text, natural language processing services or libraries to translate conversation to text, natural language understanding and language topic analysis methods, libraries, or services to analyze conversational text, natural language understanding methods, libraries, or services for word embedding/association, optical character recognition libraries to extract text from images and video frames from a screen sharing session, and image object recognition methods, libraries, or services to identify objects in video frame images from a screen sharing session.

At 202, the adaptive user interface amelioration program 150 receives shared content during a screen sharing session. As a live screen sharing session is commenced during a web conference between a sharing participant and one or more other participants, the adaptive user interface amelioration program 150 may receive an audio feed and video feed from each participant taking part in the web conference. The audio and video feeds may include metadata, such as tags and timestamps. If the audio or video feeds do not include metadata needed by the adaptive user interface amelioration program 150, the adaptive user interface amelioration program 150 may generate the needed metadata. For example, the adaptive user interface amelioration program 150 may tag an audio feed as originating from a specific participant or timestamp a video feed when a sharing participant changes to a different slide or graphical user interface view of the screen sharing session. In one or more embodiments, the video feed may include image data.

In one or more embodiments, the adaptive user interface amelioration program 150 may be configured to capture and store user profile information that includes, but is not limited to, device accessibility preferences, such as preconfigured font or image size, speaker volume preferences, microphone volume preferences, topics and keywords of interest from historical communications, and a system opt-in preference.

Then, at 204, the adaptive user interface amelioration program 150 displays the shared content on a user device display screen. The adaptive user interface amelioration program 150 may display the shared content in the course of the screen sharing session so that each participant can view the screen share as intended by the sharing participant. For example, if the sharing participant is sharing a report to an organization during a web conference, the adaptive user interface amelioration program 150 may display the report on the device display screen for each participant viewing the screen share.

Next, at 206, the adaptive user interface amelioration program 150 identifies device characteristics and user profile characteristics. In order to properly modify the viewing area of a user device display screen presenting the screen share, the adaptive user interface amelioration program 150 may determine various device statistics, including, but not limited, a type of user device (e.g., laptop, tablet, smartphone, etc.) and available screen aera. Available screen area may relate to the total area of a device display screen capable of displaying a graphical user interface generated by the user device. The adaptive user interface amelioration program 150 may identify the type of device used by a user participant in the web conference using client and/or server-side device detection methods, such as inspecting the HTTP User-Agent header.

Additionally, the adaptive user interface amelioration program 150 may identify various user profile characteristics. For example, the adaptive user interface amelioration program 150 may identify web conference participants through log-in credentials, usernames, organization names, organization roles, device IDs, etc., and retrieve user preferences, such as brightness settings, contrast settings, screen zoom settings, image quality (e.g., 1080p, 720i, etc.) settings, and any other accessibility settings, for the participants that have opted-in to sharing information with the adaptive user interface amelioration program 150.

Then, at 208, the adaptive user interface amelioration program 150 extracts topical elements within the received shared content. The adaptive user interface amelioration program 150 may utilize text-to-speech technology and natural language understanding components to translate the conversation received in the audio feed in real time and extract keywords and topics discussed between the web conference participants. For example, if the screen sharing participant is currently discussing a chart depicted on a presentation slide along with text states “Please draw your attention to the graph”, the adaptive user interface amelioration program 150 may extract the keyword “graph” and identify it as a topical element being discussed by the web conference participants.

Next, at 210, the adaptive user interface amelioration program 150 identifies screen locations of objects and text on a shared screen. The adaptive user interface amelioration program 150 may utilize the keywords, extracted in step 208, to identify screen locations of objects and text on the shared screen. The adaptive user interface amelioration program 150 may utilize optical character recognition technology, video image object detection, and image segmentation components to identify objects (e.g., images and image segments) and text and their corresponding locations on the user device display screen of a specific web conference participant in real time. For example, the adaptive user interface amelioration program 150 may identify the “graph” from the previous example is located in a specific section of the user device display screen using pixel locations or coordinates of each corner, a perimeter, or a geometric center of the identified object. In one or more other embodiments, the adaptive user interface amelioration program 150 may identify the location of an element as being the area of the element on the user device display screen.

Due to the adaptive user interface amelioration program 150 utilizing optical character recognition and other image recognition technologies to identify objects and text during a screen sharing session, the adaptive user interface amelioration program 150 may calculate a confidence score as to the accuracy of the prediction associated with an object, image, or text. Once an object, image, or text satisfies a preconfigured threshold level of confidence score, the adaptive user interface amelioration program 150 may label or tag the object, image, or text as being the element associated with the highest confidence score.

The adaptive user interface amelioration program 150 may identify, in real-time, relations between labels for objects and extracted text and their associated screen locations from screen sharing session with the keywords and topics of the current conversation. The adaptive user interface amelioration program 150 may identify the relations using natural language word embedding/association methods and/or semantics knowledge graphs of spoken concepts and labelled objects. In one or more embodiments, the adaptive user interface amelioration program 150 may store the identified relation(s) in an in-memory cache or database, such as storage 124.

Additionally, the adaptive user interface amelioration program 150 may also map the locations relative to any other objects identified on the user device display screen. For example, if the adaptive user interface amelioration program 150 identifies, through optical character recognition, that two graphs are present in a current screen share, the adaptive user interface amelioration program 150 may determine the relative locations of the two graphs to each other. The adaptive user interface amelioration program 150 may temporarily store the mapping between the objects and segments of images, their locations, labels, and confidence scores in a screen share memory repository.

Then, at 212, the adaptive user interface amelioration program 150 modifies a viewing area of shared content on the user device display screen. The adaptive user interface amelioration program 150 may modify the content based on the topical elements, the identified screen locations, the device characteristics, and the user profile characteristics. Prior to any modification, the adaptive user interface amelioration program 150 may detect the available screen area based on the identified device characteristics, including the available screen area. The adaptive user interface amelioration program 150 may extract the normal zoom or viewing size, contrast and brightness settings, and other accessibility preferences for the user from a user profile for a specific user. The adaptive user interface amelioration program 150 may modify the viewing area for a user to focus or highlight on the elements being discussed in the conversation between the web conference participants. For example, if the web conference participants are discussing a specific graph in a screen sharing session and one participant is viewing the screen share from a smartphone that makes viewing the graph difficult, the adaptive user interface amelioration program 150 may zoom into the graph and focus the image to allow the participant to more easily view the graph according to the user profile preferences. Conversely, if the adaptive user interface amelioration program 150 determines a participant is utilizing a device with adequate screen availability and quality (e.g., a large monitor displaying a high quality image), the adaptive user interface amelioration program 150 may determine that no modification or only partial modification (e.g., centering the graph being discussed on the display screen) is needed.

As another example, the adaptive user interface amelioration program 150 may modify the brightness, contrast, and/or audio volume settings to focus on or highlight either the object or text or the entire display screen in general. For example, if the user is viewing the web conference outdoors or in a brightly lit room, the adaptive user interface amelioration program 150 may determine the brightness settings of the user device should be increased to allow for better viewing of the shared content.

Additionally, in another embodiment, the adaptive user interface amelioration program 150 may modify the settings for the specific object or text being focused on and/or all elements except the object being focused on. For example, instead of zooming into the object or text, the adaptive user interface amelioration program 150 may increase the brightness of the object or text identified as being the subject of the current audio conversation and lower the brightness of the other objects and text displayed in the shared content, which overemphasizes the object or text at the subject of the web conference conversation. Similarly, the adaptive user interface amelioration program 150 may blur or lower the contrast of the elements not at the subject of the conversation while leaving the contrast of the object or text at the subject of the conversation the same.

The adaptive user interface amelioration program 150 may continuously monitor the conversation between the web conference participants and the screen share to determine if the viewing area should be modified further for one or more of the web conference participants based on changes to the topic and/or focus. The adaptive user interface amelioration program 150 may update the viewing area of the shared content should the topic and/or focus of the conversation change to a different section of the shared content.

In one or more embodiments, the adaptive user interface amelioration program 150 may consider the distance a participant observing the shared content is from the display screen of the user device. For example, the user may be seated two feet from a display screen while seated at a work desk but, if the user is utilizing a walking desk, the user may be four feet from the display screen and require the adaptive user interface amelioration program 150 to utilize more zoom when the user is observing shared content in a screen sharing session.

In one or more additional embodiments, the adaptive user interface amelioration program 150 may calculate a value associated with eye fatigue and/or eye strain. The adaptive user interface amelioration program 150 may utilize embedded or communicatively coupled photographic capture devices (e.g., cameras) to focus on a user's eyes while the user is observing the device display screen. The adaptive user interface amelioration program 150 may analyze the captured images from the photographic capture devices for characteristics associated with eye fatigue and/or eye strain, such as but not limited to, number of eye blinks, eye aspect ratio, pupil dilation, gaze detection, eye focus location, and eye openness. The adaptive user interface amelioration program 150 may be preconfigured with a weighted algorithm that predicts whether the user is experiencing eye fatigue and/or eye strain based on one or more of the characteristics. If the adaptive user interface amelioration program 150 determines the estimated value exceeds a preconfigured threshold for either the user's typical measurements or for a preconfigured average user, the adaptive user interface amelioration program 150 may determine the user is experiencing eye fatigue and/or eye strain and modify the viewing area accordingly to reduce or relieve the user eye fatigue and/or eye strain, such as changing the zoom, brightness, and/or contrast of the display screen.

Referring now to FIGS. 3A and 3B, a functional block diagram of a screen sharing session is depicted according to at least one embodiment. In FIG. 3A, content shared by a host participant is depicted as it appears on the display screen of viewing participants to the screen share. The host participant may present data relating to surveys on various weather-related categories (i.e., “uncertain how weather data creates values”, “difficulty integrating weather data into operation processes”, and “uncertain how weather data impacts decision making”) to the other web conference participants. Initially, the adaptive user interface amelioration program 150 may present all of the shared content 304 in the viewing area of the device display screen for a particular user. However, the data being presented during a specific time period may be represented in the upper-left quadrant 302 of the shared content 304 and the particular user may be viewing the shared content on a smartphone, which may make the shared content 304 difficult to view for the particular user based on user and/or device characteristics. Therefore, in FIG. 3B, the adaptive user interface amelioration program 150 may modify the viewing area of the shared content to focus in on information depicted in the upper-left quadrant 302 so that the data depicted in the upper-left quadrant 302 fills the viewing area of the user device display screen thereby making the shared content at the focus of the host participant's presentation easier to view for the particular user on the corresponding user device. For example, the adaptive user interface amelioration program 150 may zoom into the upper-left quadrant 302 so that the user can more easily see and understand the shared content on their specific device.

It may be appreciated that FIGS. 2, 3A, and 3B provide only an illustration of one implementation and do not imply any limitations with regard to how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.