COGNITIVE VIDEO SCROLLING VIEWER INTERFACE

A system and method are used to record and present a training video for a software component. The method displays the video recording. The video recording comprises: a display element of the software component, and first recorder activity data (FRAD) of recorder user interface (UI) input activity data (RUIIAD) at a FRAD time with video frame data of the video recording. The method further comprises receiving, during the displaying, viewer UI actions from a UI of the viewing device and converting them to viewer UI input activity data based on viewer interactions with the display element of the software component. Then, first viewer activity data (FVAD) is determined at a FVAD time of the video frame that corresponds to the FRAD, where the FRAD time and the FVAD time differ. Responsive to the determining, the method moves the video recording from the FVAD time to the FRAD time.

BACKGROUND

Disclosed herein is a system and related method for a viewer interface, and, in particular, a cognitive video scrolling viewer interface. Videos often have a horizontal scroll bar that may be used by a viewer to control the video being played. Popular online learning may involve one or more videos, and different types of learning content may be recorded in video—these may be played for video-based learning.

SUMMARY

Disclosed herein is a computer implemented method for producing, by a recorder, a video recording about a software component, using a recording device. The method comprises using a processor for receiving, during a recording operation for training a viewer to operate the software component, video frame data for the video recording of the recorder using a user interface (UI) interacting with the software component. The method further comprises receiving, during the recording operation, recorder UI actions from a UI of the recording device and converting them to recorder UI input activity data (RUIIAD) based on the recorder interactions between the recorder and a display element of the software component. First recorder activity data (FRAD) of the RUIIAD is associated at a FRAD time RT1with the video frame data of the video recording. Finally, the video recording that comprises the video frame data and the RUIIAD is saved to a recording medium.

Also disclosed herein is a computer-implemented method for presenting, to a viewer, a video recording about a software component, using a viewing device. The method comprises using a processor for displaying, during a viewing operation for training the viewer to operate the software component on a display device, the video recording. The video recording comprises: a display element of the software component, and first recorder activity data (FRAD) of recorder user interface (UI) input activity data (RUIIAD) at a FRAD time RT1with video frame data of the video recording. The method further comprises receiving, during the displaying, viewer UI actions from a UI of the viewing device and converting them to viewer UI input activity data (VUIIAD) based on viewer interactions between the viewer and the display element of the software component. The method further comprises determining first viewer activity data (FVAD) of the VUIIAD at a FVAD time VT1of the video frame that corresponds to the FRAD, where the FRAD time VR1and the FVAD time VT1differ. Responsive to the determining, the method moves the video recording from the FVAD time VT1to the FRAD time RT1.

Also disclosed herein is a computer-implemented system that includes a recording device and a viewing device, where the devices comprise memory and processors to implement the above-described methods.

DETAILED DESCRIPTION

The following general acronyms may be used below:

General

Further, the phrases “at least one”, “one or more,” and “and/or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and/or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. In other words, “at least one of”, “one or more of”, and “and/or” mean any combination of items and number of items may be used from the list, but not all of the items in the list are required. The item may be a particular object, a thing, or a category. Additionally, the amount or number of each item in a combination of the listed items need not be the same. For example, in some illustrative examples, “at least one of A, B, and C” may be, for example, without limitation, two of item A; one of item B; and ten of item C; or0of item A; four of item B and seven of item C; or other suitable combinations.

Furthermore, the term “automatic” and variations thereof, as used herein, refers to any process or operation done without material human input when the process or operation is performed. However, a process or operation can be automatic, even though performance of the process or operation uses material or immaterial human input, if the input is received before performance of the process or operation. Human input is deemed to be material if such input influences how the process or operation will be performed. Human input that consents to the performance of the process or operation is not deemed to be “material.”

Although specific embodiments are illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement, which is calculated to achieve the same purpose, may be substituted for the specific embodiments shown. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.

Data Processing System in General

Cognitive Video Scrolling Viewer Interface

The following application-specific acronyms may be used below:

TABLE 2Application-Specific AcronymFRADfirst recorder activity dataFVADfirst viewer activity dataRUIIADrecorder user interface input activity dataUIuser interfaceVAEvideo analysis engineVRvirtual realityVUIIADviewer user interface activity data

FIG.2is a block diagram showing an example according to various embodiments of a recording device200, which may be implemented, e.g., on a computer101and within a computing environment100. The recording device200may be used to produce software component training videos constituting video recordings250that are then stored on a recording medium15.

FIG.3is a block diagram showing an example according to various embodiments of a viewing device300, which may also be implemented, e.g., on a computer101and within a computing environment100. The viewing device300may be used to present/view the video recordings250stored on the recording medium15that were produced by the recording device200.

FIG.4is a screen shot of a physical display as it displays an example of recorder10and viewer20interactions with the recorded software component210and a video playback of the video recording250respectively. For the sake of simplicity,FIG.4is used to illustrate the physical display of both the recording device200and the viewing device300. However, there is one significant difference between its illustration of a display for the two devices200,300: for the recording device200, there is no video viewer, and the display of the spreadsheet application and formatting dialog box are created by execution of the actual spreadsheet application itself; for the viewing device300, the video viewer is present, and the display of the spreadsheet application and formatting dialog box are within the video recording itself. Hence, there is no need for the application software to be present on the viewing device300. Reference toFIG.4will be interleaved with the discussion ofFIG.2, the recording device200, and the discussion ofFIG.3, the viewing device300.

InFIG.2, a user who is a recorder10executes a software component210. In the example application shown inFIG.4, the software component210is a spreadsheet410. The recorder10may interact with the software component210via a user interface250, which may comprise input devices206and output devices, such as the display208. The input devices206may include devices such as a keyboard that has keys pressed, a mouse that is moved on a surface and has buttons that pressed for selection and other actions, trackball that is rotated and has buttons that are pressed for selection and other actions, a touch screen that is tapped or has one or more fingers dragged to perform various functions, an eye tracker in a virtual reality (VR) headset where voice or gestures are used to select displayed items, a microphone for audio input, such as voice commands, or any form of pointing device. The output devices may also include a VR headset, monitor, speaker, and the like. In the example application, one input device is a pointing device that is a mouse.

In the example, the recorder10is providing a training lesson on using various aspects of the spreadsheet410. The recorder10is recording what is being shown on the display208into a video recording250that is saved to a non-volatile recording medium15. In a traditional training video, the video recording250would only comprise video frame data254. However, in various embodiments disclosed herein, data for various recorder activity, the recorder UI input activity data (RUIIAD)240(such as the first recorder activity data (FRAD) RT12401) are stored with the video recording250as well.

In a focus operation, the recorder10is teaching the viewer20how to bold the text in a cell. As can be seen inFIG.4, the spreadsheet410has three cells, and a middle cell, cell A1315is currently selected, shown as shaded in the FIG. The recorder10has initiated a cell formatting function, and a display element212in the form of a dialog box412appears on the display208. In the example, and to illustrate various embodiments better, the recorder10, with the display as shown inFIG.4, pauses before actually pressing the bold button415so that she can explain all of the various format features that are available. This explanation continues for two minutes, and then the recorder10finally presses the bold button415resulting in the dialog box412disappearing and the text within cell A2315becomes bolded.

FIG.2illustrates the use of a video analysis engine (VAE)220that receives, e.g., the user interactions with the display208using the input device206. The VAE220may also receive input from (or related to) the display208about the software component210and any display elements212that are being presented by the software component210as well as the recorder10activities and interactions with the display208via one or more input devices206. The display elements212may be, e.g., a dialog box, a menu, a drop-down menu, a list box, a push button, a radio button, a checkbox, and any other display mechanism for soliciting user input. The display elements212may comprise any display elements212in combination, and may be nested within other display elements212. For example, and OK pushbutton may be nested within a dialog box and used for confirmation of some activity.

It then converts the data into the RUIIAD240and synchronizes it with the video frame data254. As shown inFIG.2, a plurality of recorder10activities may create different RUIIAD240at different times, e.g., RT12401, RT22402, RTN240N. The recorder10actions may include, for example, selecting from a drop-down list, scrolling, selecting a menu option, etc. The video analysis engine performs post processing of the video being recorded and identifies appropriate frames and the positions where the action is to be created, so that, while watching video, the viewer can also perform the same action.

The VAE220may identify a pattern of cursor movement, including, e.g., movement direction, click action, etc., and, accordingly, add data associated with appropriate action on the video recording250so that, while watching the video recording250, the viewer20may interact with the displayed software component210,310in a similar manner.

Like cursor and mouse movements, in which the cursor movement pattern is tracked on the screen, the VAE220may identify voice commands used by the recorder10while the video recording250is being created, and, accordingly, adds appropriate action(s) on the video recording250so that while watching the video recording250, the viewer20may interact with original content using the same voice commands.

Using the example ofFIG.4, the recorder10presses the bold button415on the format dialog box412(the display element212), and information about the button press is stored as the FRAD2401at time T1, which is associated with time T1in the video frame data254. The VAE220may determine the X, Y screen coordinates of the input device206pointer when a select activity is performed. The VAE220may then determine which display element212of which software component210and which software component activity the select activity corresponds to. For example, the VAE220may determine that the X, Y screen coordinates correspond to the bold button415of the format dialog box412for the spreadsheet410. In some embodiments, the X, Y coordinates may be relative to various display elements212of the software component210.

The VAE220may also identify a previous video frame of the video frame data254to find what action is performed by the positioning input device206, e.g., a mouse pointer, which has changed the video content. In various embodiments, the VAE220may identify the relative position of the mouse pointer with respect to a particular recorded video frame, and may identify a location on the video frame where the mouse action is performed.

In various embodiments, the VAE220may also track the pointer device movement pattern and pattern of change in a video frame of the video frame data254. The VAE220may identify actions, position of the action on the video frame, and the same action will be considered as an overlaid action, creating the RUIIAD240data. The VAE220may identify the video frames where the change in the video content is present, and may identify an appropriate action and the position of the scroll bar.

The video recording250may have frame identified actions assigned on different portions of the video frames of the video frame data254. In some embodiments, the VAE220overlays actions on exact positions of the frames, on a recognized object. The VAE220may identify an object boundary where the recorder10action was performed. In some embodiments, the VAE220may overlay the identified actions on the frame and the actions may be transparent.

Upon completion, the video recording250may be stored in the recording medium15where it may be accessed by the viewer.

FIG.3is a block diagram of the viewing device300that the viewer20may use to view the video recording250stored on the recording medium15. In some embodiments, while the video recording250is being playing, the viewer20may play the video recording250as per normal playing of the video, and may control the video recording250via the scroll bar.

However, in the novel viewing device300, while watching the video recording250, the viewer20may use the positioning device on the video content while it is being played.

The video viewer402may identify where the viewer20has performed their action. In this case, the touch screen enabled device may identify the touch position on the display308where the viewer tapped the display308. The VAE220may send information to the video viewer402, and the VAE220may map the position where the touch action is performed. Based on the touch or mouse action on the video content, the VAE220may control the movement of the video scroll bar. Based on the viewer's20action on the video screen that is being played or based on voice commands of the viewer20that match the voice commands of the recorder10, the video scroll bar may dynamically be changed to the appropriate position. For matching voice commands, any known voice recognition software or hardware may be used.

In more detail, the video recording250contains both the RUIIAD240and the video frame data254that was created by the recorder10. As noted above, instead of the viewer20interacting with the actual software component210and display element212as the recorder10did, the viewer20interacts with the recorded software component310and the recorded display element312.

The VAE220now takes on a different role with the viewing device300than it had with respect to the recording device200. AlthoughFIG.2andFIG.3show the VAE220as being the same and using the same reference number, they could actually be two separate components that operate in a different way, depending on whether it is a recording device VAE or a viewing device VAE. On the viewing device, the VAE220reads the data from the video recording250instead of writing to it. The VAE220also receives user input device306activity as the viewer20interacts with the display308and creates viewer UI activity data (VUIIAD)340, which may be similar to the RUIIAD240. The input device306may constitute similar devices to recording devices200input devices206. In this case, however, the VUIIAD340is coordinated with what is being displayed on the video viewer402, and not what is being produced by execution of the software component210itself. As noted above, the viewing device300need not have the actual software component210installed to view the video recording250.

As shown in the example ofFIG.4, the viewer20is watching the video recording250on their display308within a video viewer application402, which shows the recorded software component310, i.e., the recorded spreadsheet410, and the recorded display element312, i.e., the recorded format dialog box412. As the viewer20watches the video recording250, he realizes that the recorder10is going to discuss each and every format option available for formatting a cell. Since the viewer20is only interested in seeing the bolding operation, the viewer20wishes to skip the other two minutes of dialog on the video recording and skip directly to the point at which the bold button415is selected.

In this case, the viewer20clicks the mouse pointer on the bold button415, just as the recorder10did when interacting with the spreadsheet410. In a conventional viewing device, such a mouse click would have had no effect, since a select on a pointing device within an area of video display conventionally has no effect. The viewer20is not able to perform the actions performed by the recorder10, e.g., the viewer cannot click on the dialog box button but must simply watch the recorder10run through various other activities. In this situation for a conventional viewing device, if the viewer20wants to see the effect of pressing the button, the viewer must scroll through the video until the recorder10presses the button. While watching the video, the viewer is not be able to perform the actions of the recorder on the displayed content. For the viewer, only the scroll bar associated with the video view display of the video recording is available.

However, in various embodiments disclosed herein, the VAE220, is able to create a VUIIAD element (first viewer activity data FVAD) VT13401by selecting the bold button415within the video recording250. The VAE220is able to coordinate this selection information, i.e., using screen or application X, Y coordinates with a similar activity within the RUIIAD240.

In other words, the FVAD VT13401information is determined to be similar to the FRAD RT12401information (based on coordinates and higher-level information, such that both correspond to selection of a bold button415within a format412dialog box). Based on this similarity determination by the VAE220, the VAE220may advance the viewing position of the video recording250to a point at which the similar activity during recording took place (i.e., of the FRAD RT12401). In this way, the viewer20may jump or advance directly to a position (of the FRAD RT12401) in the video recording250in which the viewer's action would have taken them if they were interacting with the actual software component210. Thus, the viewer20does not have to watch portions of the video recording250that are unimportant to him. Advantageously, according to various embodiments disclosed herein, the viewer20may perform the same actions upon viewing the recorded video250as the recorder10performed while creating the video recording250.

Such viewer20activities could be matched with any of the RUIIAD events (RT22402, RTN240N). If there is no matching RUIIAD events, the VAE220could simply ignore the viewer20activities, or, alternately, could provide some form of indication that no activity (or corresponding RUIIAD) for a particular interaction was recorded.

FIG.5is a flowchart of an example video recording process500in accordance with some embodiments described herein. On a recording device200, in operation502, video frame data254is received based on the output produced by the executing software component210as well as any output shown based on recorder10user interface205activity, such as positioning of a mouse arrow, keystrokes displayed in a field and the like. In operation504, while the video frame data is being received, the RUIIAD240is also received, based on the recorder's10interactions with the software component.

In operation506, the RUIIAD240is associated with the corresponding video frame data254to produce the video recording250by the VAE220. In operation508, the video recording250is stored in a non-volatile recording medium15.

FIG.6is a flowchart of an example video viewing process600in accordance with some embodiments described herein. On a viewing device300, in operation602, a video recording250is displayed on a display308. In operation604, as the viewer20is viewing the video recording250, the VAE220receives viewer UI actions and converts them into VUIIAD340.

In operation606, the VAE220determines if there is a RUIIAD element RT12401that matches the VUIIAD element3401VT1. If so, in operation608, the video recording is moved to a time corresponding to the RUIIAD element RT12401

Various embodiments may offer, as a service, a way to cognitively understand and correlate user interactions in the real world on the video with actions within the video, and dynamically re-route viewing to other sections of the video. Various embodiments serve to understand the video rather than merely summarize it, and provide capabilities of self-referencing and learning.

Technical Application

The one or more embodiments disclosed herein accordingly provide an improvement to computer technology. For example, an improvement to providing a user interface for training videos displayed on a computer display helps improve the efficiency in operating training videos provided for computer users.