Remote technology assistance through dynamic flows of visual and auditory instructions

Systems, methods, and computer program products to perform an operation comprising receiving, by a first device, metadata describing a first action of a plurality of actions performed on a second device, identifying a target object of the first action based on the metadata describing the first action, and outputting, by the first device: a sequence of images depicting performance of the first action on the second device, a textual instruction specifying how to perform the first action on the first device proximate to the target object, and an audio instruction specifying how to perform the first action on the first device.

BACKGROUND

The present invention relates to computer software, and more specifically, to computer software which provides remote technology assistance through dynamic flows of visual and auditory instructions.

Users often have difficulty using computers and related technology. As a result, many people cannot enjoy the benefits provided by computer technology in everyday life. While some users can learn via documentation and other tutorials provided by hardware and software providers, these tools are static in nature, and often do not cover all possible use cases. In fact, many users cannot learn how to use technology via these tools. Instead, these users require personal, focused assistance to learn. Such personal assistance is time consuming and energy intensive. Furthermore, users have different paces of learning and best modes of learning, rendering a “one-size fits all” assistance approach as relatively ineffective.

SUMMARY

In one embodiment, a method comprises receiving, by a first device, metadata describing a first action of a plurality of actions performed on a second device, identifying a target object of the first action based on the metadata describing the first action, and outputting, by the first device: a sequence of images depicting performance of the first action on the second device, a textual instruction proximate to the target object, wherein the textual instruction specifies how to perform the first action on the first device, and an audio instruction specifying how to perform the first action on the first device.

In another embodiment, a system comprises a processor and a memory storing instructions, which when executed by the processor, performs an operation comprising receiving, by the system, metadata describing a first action of a plurality of actions performed on a first device, identifying a target object of the first action based on the metadata describing the first action, and outputting, by the system: a sequence of images depicting performance of the first action on the first device, a textual instruction proximate to the target object, wherein the textual instruction specifies how to perform the first action on the system, and an audio instruction specifying how to perform the first action on the system.

In another embodiment, a computer-readable storage medium has computer-readable program code embodied therewith, the computer-readable program code executable by a processor to perform an operation receiving, by a first device, metadata describing a first action of a plurality of actions performed on a second device, identifying a target object of the first action based on the metadata describing the first action, and outputting, by the first device: a sequence of images depicting performance of the first action on the second device, a textual instruction proximate to the target object, wherein the textual instruction specifies how to perform the first action on the first device, and an audio instruction specifying how to perform the first action on the first device.

DETAILED DESCRIPTION

Embodiments disclosed herein allow users to remotely assist other users to learn how to use computer technology, such as software applications, operating systems, hardware features, peripheral devices, and the like. Generally, a first user may perform a series of actions on a first computing device to help another second user learn how to perform the series of operations on a second computing device. Embodiments disclosed herein may record the series of actions as they appear on the display of the first computing device (e.g., a screen capture of the display of the first computing device), and capture metadata describing each action on the first computing device. The metadata and recorded series of operations on the display may be transmitted to the second device to assist the second user. In at least one embodiment, related operations are grouped together and sent to the second device. Once received on the second device, a tutorial describing the series of actions may be outputted to show the second user how to perform the series of actions. The tutorial outputted on the second device may include audio and visual components, such as a text-to-speech conversion of each step of the process and a step-by-step visual tutorial of each action performed by the first user.

If the series of actions are specific to the first device (e.g., are not directly compatible with the second device), embodiments disclosed herein may translate the series of actions to a format compatible with the second device. Furthermore, if the first and second users do not speak the same language, embodiments disclosed herein may provide the tutorials in a language understood by the second user.

FIG. 1Aillustrates an example system100which provides remote technology assistance through a dynamic flow of visual and auditory instructions, according to various embodiments. As shown, the system100includes two computing devices101,102. The computing devices101,102are representative of any type of computing device, such as a desktop computer, laptop computer, smartphone, tablet computer, portable gaming device, and the like. Generally, a user of the computing device101may assist the user of computing device102.FIG. 1Adepicts an example action (or operation) performed in the assistance process, where the user of the computing device101opens an application by clicking an icon104depicted on a display103of the computing device101.

When the user clicks the icon104, the computing device101captures the event, and generates the action information106. The action information106includes metadata describing the captured event (e.g., navigating to the screen where the icon104is present, and the user clicking on the icon104to open the application). The action information106may include actual system calls (e.g., events, function calls, event handlers, and the like) performed by the operating system (not pictured) of the computing device101, as well as metadata describing each action performed by the operating system. For example, the metadata may describe the icon104, which may include a name of the icon, a target application, textual and/or voice input describing how to perform the action, and other associated metadata. The computing device101may then transmit the generated action information106to the computing device102via a network (not pictured).

As part of the assistance process, the computing device101may record the output of the display103, such as when the user navigates to the screen where the icon104is located, and the user clicking on the icon104. The computing device101may generate an animated graphics interchange format (GIF) file113depicting the steps performed on the computing device101. Although a GIF is used herein as a reference example, a video or other sequence of images may be used. The computing device101may then transmit the GIF113to the computing device102. The computing device102may then continuously loop the animated GIF113on the display103to show the user of the computing device102the actual steps performed by the user of the computing device101to open the application.

In some embodiments, the action information106is specific to the computing device101, and not compatible with the computing device102. For example, the computing device101may execute a different operating system than the computing device102, and the platform-specific system calls stored in the action information106may not be understood by the operating system of the computing device102. In such embodiments, the action information106needs to be translated to a format compatible with the computing device101. Therefore, the computing device101, the computing device102, and/or a server (not pictured) may translate the action information106to a format compatible with the computing device102based on one or more translation rules.

Once the action information106is compatible with the computing device102, a series of visual and auditory assistive cues may be generated for output on the display103of the computing device102. Generally, the computing device102may process the action information to identify the icon104on the computing device102, and output the icon104on the display103. In addition, the computing device102may identify the step of opening the application associated with the icon104in the action information106, and generate an indication conveying that the user of the second computing device needs to select the icon104to open the associated application. As shown, the computing device102has generated a text bubble105which is outputted on the display103, and instructs the user to “click here to open app.” As previously indicated, the assisting user may provide the text specifying to “click here to open app.” By identifying the icon104, and outputting the text bubble105, the user of the second device may be able to open the desired application.

Furthermore, as shown, the computing device102outputs a sound clip130via a speaker (not pictured) to further assist the user. Generally, in the example depicted inFIG. 1A, the sound clip130is an auditory instruction which tells the user to select the icon104to open the associated application. In at least one embodiment, the user of the computing device101may speak instructions for the user of the computing device102, which are recorded and stored as the sound clip130. In other embodiments, one or more of the computing devices101,102(or a different computing device) may generate the sound clip130based on the actions described in the action information106. For example, the action information106(and/or the text of the text bubble105) may be converted to speech using a text-to-speech algorithm. As another example, the computing device102may store a plurality of sound clips that are enriched with metadata. The computing device102may match metadata of the action information106to the metadata of the stored sound clips to select and output appropriate auditory instructions.

Therefore, as shown inFIG. 1A, the user of the computing device102has the text bubble105, the animated GIF113, and the sound clip130to assist them in opening the application associated with icon104. In the event that the user of the computing device102speaks a different language than the user of computing device101(or is more comfortable with a different language than the language used by the user of computing device101), the text outputted in the text bubble105and the speech outputted as part of the sound clip130may be converted to a different language. Doing so allows users to receive assistance in a preferred language of their choosing (which may be specified in a user profile associated with each user).

In some embodiments, an “offline” tutorial session may be implemented, where the user of the computing device101performs all of the required actions at once. In such embodiments, the actions may be grouped into related actions, and the associated assistance tools (e.g., the text bubble105, the animated GIF113, and the sound clip130) may be outputted as the user of the computing device102successfully completes each group of actions. Once the user completes a group of actions, the next set of assistive tools may be “unlocked” and outputted on the computing device102to help the user complete the remaining actions.

FIG. 1Billustrates an embodiment where the user of the computing device102has successfully opened the application associated with the icon104. As shown, the display103of device101now depicts an input text box107and a submit button110as part of a graphical user interface (GUI) of the opened application. The user of the computing device101may then select the text box107and provide text input to the text box107. The computing device101may capture each of these actions, and generate corresponding action information106. The action information106may therefore include event information from the application (e.g., an application call to activate the text box107) and operating system calls (e.g., event handlers to display a keyboard and provide text input). Furthermore, the computing device101may continue to capture the output of the display103, and generate the animated GIF113which includes the latest actions performed by the user of the computing device101(e.g., selecting text box107and typing in the text box107). The computing device101may again transmit the action information106and the animated GIF113to the computing device102.

As shown, the display103of the computing device102includes the input text box107and the submit button110of the GUI of the opened application. The display103also includes two text bubbles108,109. The text bubbles108,109may again be generated based on the updated action information106. As shown, the text bubble108instructs the user of the computing device102to “Input Text Here,” while the text bubble109includes a Spanish translation of the text in the text bubble108. In at least one embodiment, however, only one of the text bubbles108,109may be outputted to the user. In at least one embodiment, the action information106includes a tooltip associated with the text box107. The tooltip may include instructions that are outputted when the user hovers their mouse over an object (e.g., the text box107). The tooltip information may be leveraged to assist the user, for example, in generating the text of the text bubble108.

Furthermore, the computing device102may again output a sound clip130which generally instructs the user to input text in the text box107. As inFIG. 1A, the sound clip130may be selected from a stored library of sound clips, generated by the user of computing device101, or generated based on a text-to-speech conversion of the text of text boxes108,109.

As inFIG. 1A, the computing device101may send feedback121to the computing device120. The feedback121may indicate whether the user of computing device has successfully completed the actions in the action information106(e.g., activated the text box107and entered text in the text box107). The computing device101may use any means to determine whether the user has successfully completed the actions, such as by recording system calls made by the application, examining objects on a system stack, and analyzing image data of a screenshot (to determine whether text exists in the text box).

FIG. 1Cdepicts an embodiment where the user of computing device102has successfully entered text into the text box107. As shown, the user of computing device101may then click on the submit button110to submit the input to the application for further processing. The computing device101may record the user's action of clicking the submit button110to generate an updated animated GIF113, which visually depicts the action of clicking the submit button110. The updated animated GIF113may then be transmitted to the computing device102, where it is again outputted on the display103to assist the user of computing device102.

As shown, the computing device101also generates updated action information106describing the click of the submit button110. The updated action information106may specify attributes of the submit button110(e.g., an object name, type, and other parameters) that the computing device102may use to properly identify the submit button110, in the event there are multiple objects outputted on the displays103by the application. In the event that the user of computing device102has navigated away from the application including the submit button110, the computing device102may use the attributes stored in the action information106to identify the correct application, and display the application (and the submit button110) on the display103.

As shown, the computing device102may generate and output text bubbles111,112specifying to click the submit button110. Furthermore, the computing device102may output a sound clip130which generally instructs the user to click the submit button110to submit the text they entered in text box107. The sound clip130may include instructions in a plurality of different languages. Additionally, the computing device102outputs the animated GIF113showing the user clicking the submit button110. In at least one embodiment, the animated GIF113is cumulative in that all of the previous actions performed by the user of the computing device101are depicted.

As inFIGS. 1A-1B, the computing device102may track the actions of the user, and send feedback to the computing device101. If the user successfully completed the actions included in the action information106, the user of computing device101may perform any additional actions to assist the user. In the “offline” embodiment, the computing device102may determine whether more actions are stored in the action information106. If more actions remain, the computing device102may output the next set of instructional tools on the display103.

FIG. 2is a block diagram illustrating a system200which provides remote technology assistance through a dynamic flow of visual and auditory instructions, according to one embodiment. The networked system200includes a computer202. The computer202may also be connected to other computers via a network230. In general, the network230may be a telecommunications network and/or a wide area network (WAN). In a particular embodiment, the network230is the Internet.

The computer202generally includes a processor204which obtains instructions and data via a bus220from a memory206and/or a storage208. The computer202may also include one or more network interface devices218, input devices222, and output devices224connected to the bus220. The computer202is generally under the control of an operating system (not shown). Examples of operating systems include the UNIX operating system, versions of the Microsoft Windows operating system, and distributions of the Linux operating system. (UNIX is a registered trademark of The Open Group in the United States and other countries. Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.) More generally, any operating system supporting the functions disclosed herein may be used. The processor204is a programmable logic device that performs instruction, logic, and mathematical processing, and may be representative of one or more CPUs. The network interface device218may be any type of network communications device allowing the computer202to communicate with other computers via the network230.

The storage208is representative of hard-disk drives, solid state drives, flash memory devices, optical media and the like. Generally, the storage208stores application programs and data for use by the computer202. In addition, the memory206and the storage208may be considered to include memory physically located elsewhere; for example, on another computer coupled to the computer202via the bus220.

The input device222may be any device for providing input to the computer202. For example, a keyboard and/or a mouse may be used. The input device222represents a wide variety of input devices, including keyboards, mice, controllers, and so on. Furthermore, the input device222may include a set of buttons, switches or other physical device mechanisms for controlling the computer202. The output device224may include output devices such as monitors, touch screen displays, and so on.

As shown, the memory206contains a plurality of applications203and an assistance application212. The applications203are representative of any type of computing application, such as web browsers, email applications, word processors, and the like. The assistance application212is generally configured to facilitate remote user assistance with visual and auditory instructions. For example, a user of the computing device250may execute an instance of the assistance application212to assist a user of the computer202in using one of the applications203. Generally, the assistance application212captures actions performed by an assisting user and generates action information106describing the actions performed by the assisting user. The assistance application212may then transmit the assistance information106to the target device used by the user requiring assistance (e.g., the computer202). The assistance application212may also create an animated GIF based on the display of the assisting device which records the on-screen actions performed by the assisting user, and outputs the animated GIF for the user requiring assistance. Furthermore, the assistance application212may generate text bubbles, sound clips, and any other type of auditory and/or graphical assistance tools based on the action information106. The assistance application212may also leverage the platform translation216to convert platform-dependent actions to global actions that can be understood by other devices. The assistance application212may further translate the assistance tools to a preferred language of the assisting user based on the language translation215. The assistance application212may monitor the actions performed by the user receiving assistance, and determine whether the user successfully completes each action in the action information106. Doing so allows the assistance application212to provide stages of assistance that guide the user through a multi-stage process when learning how to use computer technology. Furthermore, when assisting a user, the assistance application212may group related actions together (e.g., identifying a web browser icon and clicking the icon to open the web browser), so as to not overwhelm the user with too many instructions.

As shown, the storage208contains the action information106, profiles213, objects214, a language translation215, a platform translation216, and a text-to-speech217. As described above, the action information106includes metadata describing actions performed by an assisting user. The action information106may be stored on the assisting device and/or the target device (the device operated by the user requiring assistance). The action information106may include real-time actions performed by an assisting user, and may also include a complete set of previously recorded actions to assist users in “offline mode.” In offline mode, the assisting user need not be active on the assisting computing device. Instead, the local assistance application212presents the actions in sequence to the user requiring assistance until all actions are completed. In some embodiments, the action information106includes predefined validation points, which are used to define a set of steps that must be performed by a user before a new set of instructions are outputted. For example, a validation point associated withFIG. 1Amay require the user to click on the icon104prior to displaying the assistance depicted inFIG. 1B. The assistance application212may further use the validation points in determining whether the target user has successfully completed an action.

The profiles213include user profiles for a plurality of users. The user profiles213may include various settings and parameters for each user, such as preferred languages, preferred methods of instruction (e.g., sound clips or text-based instructions), and the like. The objects214include auditory and visual objects that are used to provide user assistance. For example, the objects214may include sound clips, animated GIFs, text bubbles, tooltips, and the like. The objects214may include templates and/or completed objects. The objects214may include predefined objects as well as objects dynamically created by the assistance application212.

The platform translation216includes mappings and rules that allow the assistance application212to convert platform-dependent information in the action information106from one platform to another. For example, the platform translation216may include entries and rules for mapping operating system calls in operating system X to corresponding operating system calls in operating system Y. The text-to-speech217is a data store used by the assistance application212to generate audio clips based on text. For example, the assistance application212may use the text-to-speech117to generate audio based on the action information106and/or objects214.

The computing devices250are representative of any type of computing devices, such as the computing devices101,102. As shown, the assistance application212and a set of applications203execute on the computing devices250. The instance of the assistance application212executing on the computer202and the computing devices250can be used by either an assisting user or a target user receiving assistance.

FIG. 3is a flow chart illustrating an example method300to provide remote technology assistance through a dynamic flow of visual and auditory instructions, according to one embodiment. As shown, the method300begins at block310, where an assisting user performs actions on an assisting device. In at least one embodiment, the assisting user may load the assistance application212prior to performing the actions at block310. Example actions may include the assisting user opening an application store on a mobile device, searching for an application, specify to install the application, and accept license terms to install the application. At block320, the assistance application212may optionally translate the actions performed on the assisting device to a format understood by the target device. For example, if the target device uses a different application store than the assisting device, the assistance application212may replace the application store application associated with the source device with the application store application associated with the target device.

At block330, the assistance application212may transmit action information generated at block310for each action performed on the assisting device to the target device. At block340, the assistance application212may optionally translate assistance information (e.g., text or audio stored in the objects214) to a target language based on the preferred languages specified in the profile213of the target user. At block350, described in greater detail with reference toFIG. 5, the assistance application212on the target device may output assistance on the target device. For example, in a first set of instructions, the assistance application212may identify the icon for the application store on the target device, and cause the target device to display the icon for the application store. The assistance application212may also create a text bubble telling the user to click on the icon, while a sound clip tells the user to click on the icon. An animated GIF may also show the assisting user's actions in navigating to the application store icon, and selecting the icon to open the application store, and so on.

At block360, the assistance application212may validate the actions on the target device. Generally, the assistance application212may record actions performed by the user on the target device, and determine whether these actions satisfy a sequence of events listed in the action information106. If the user successfully completes the required actions, the assistance application212may output the next set of instructions, including audio and visual tools. At block370, a user and/or the assistance application212may optionally provide feedback based on the validated actions. For example, an instance of the assistance application212executing on the target device may provide an indication that the target user has or has not successfully completed the current actions. As another example, the target user may specify preferences as to the outputted assistance. For example, the target user may indicate that a sound clip including instructions was preferred to a text bubble.

FIG. 4is a flow chart illustrating an example method400corresponding to block310to perform actions on an assisting device, according to one embodiment. As shown, the method400begins at block410, where the assistance application212records the display of the assisting device as the assisting user performs actions on the assisting device. Doing so allows the assistance application212to generate an animated GIF showing the steps taken by the assisting user, which may be useful in instructing the target user. At block420, the assistance application212executes a loop including blocks430-470for each action performed on the assisting device. At block430, the assistance application212may determine a target object of the current action. In at least one embodiment, the target object of the current action is specified in the action information106for the current action received from the assisting user. The target object may be any type of object, such as an icon, uniform resource locator (URL), form button, application widget, input element, and the like. At block440, the assistance application212may identify a type of the current action, such as opening a file, saving a file, opening an application, submitting a form, requesting remote content, and the like. At block450, the assistance application212may generate additional metadata for the current action, such as system calls generated as part of executing the current action, a timestamp of the current action and/or system calls, and the like. At block460, the assistance application212may store the information gathered at blocks430-460in the action information106for the current action. The assistance application212may further specify other information used to facilitate tutorial sessions, such as assisting and target users associated with the action information106, applications associated with the action information106, versions of the applications, and the like.

At block470, the assistance application212determines whether any additional actions remain. If more actions remain, the assistance application212returns to block420. Otherwise, the assistance application212proceeds to block480, where the assistance application212generates the animated GIF based on the recorded display. At block490, the assistance application212may optionally group related actions, and define validation points for the grouped actions. If the assistance application212groups related actions, the assistance application212may update the action information106to reflect the groupings and/or validation points. Doing so allows the assistance application212on the target device to output assistance in smaller steps so as to avoid overwhelming the target user.

FIG. 5is a flow chart illustrating an example method500corresponding to block350to output assistance on a target device, according to one embodiment. As shown, the method begins at block510, where the assistance application212executes a loop including blocks520-590for each action performed by the user of the assisting device and received by the target device. At block520, the assistance application212optionally outputs an animated GIF for the current action on the display of the target device. As previously indicated, the animated GIF may reflect one or more actions performed by the assisting user. Therefore, the animated GIF outputted at block520may depict the screen of the assisting user performing the current action and/or additional actions.

At block530, the assistance application212optionally outputs audio instructions on a speaker of the target device. In one embodiment, the assisting user may speak instructions which are recorded by the associated computing device, and sent to the target device for outputting. For example, the user may speak “click on the icon to open the application,” which is recorded as a digital audio file, transmitted to the target device, and outputted for the user needing assistance. In another embodiment, the audio instructions are stored in the objects214and selected by the assistance application212based on associations with the current action performed by the assisting device. In some embodiments, the assistance application212may output more than one audio instruction. For example, if the user does not correctly perform the current action after hearing audio instructions from the assisting user, the assistance application212may output a related audio file from the content214to further assist the user.

At block540, the assistance application212translates the current action to an action compatible with the target device upon determining that the current action is not compatible with the target device. For example, a command specific to web browser Y may be translated to a command understood by web browser Z, where the assisting device executes web browser Y and the target device executes web browser Z. At block550, the assistance application212may identify the target object of the current action and output the target object on the target device. For example, if a user is unaware as to where an application icon is located, the assisting user may submit a command showing the user how to open the application. The assistance application212may identify metadata attributes (e.g., name, location, and target object) of the icon in the action information106, and use these attributes to identify the corresponding icon on the target device.

At block560, the assistance application212may output a visual indicator proximate to the target object of the current action. For example, the visual indicator may be a text bubble, graphic, text enhancement, and the like, which draws the user's attention to the target object. At block570, the assistance application212determines whether the user of the target device successfully performs the current action. To determine whether the user successfully performed the current action, the assistance application212may monitor the system stack to identify a corresponding function specified in the action information106for the current action, identify function calls invoked by the application and/or operating system, and the like. At block580, the assistance application212may store an indication of whether the user successfully performed the current action on the target device. In at least one embodiment, if the user does not successfully perform the current action, the assistance application212may return to block520to output additional assistance to the user in the form of visual and audio indicators. At block590, the assistance application212determines whether additional assistance actions remain. If more actions remain, the assistance application212returns to block510, otherwise the method500ends.

Advantageously, embodiments disclosed herein provide techniques to remotely provide assistance to users of computing devices. Embodiments disclosed herein provide visual cues in the form of animated GIFs depicting the necessary steps needed to complete desired computing actions, visual indicators which identify target objects and required user actions for the target objects, and audio instructions that a user can listen to for assistance. Furthermore, the assistance may be provided in any number of languages. Advantageously, embodiments disclosed herein improve the field of assistive technology by facilitating assistance from remote users (or offline assistance) and providing the necessary tools required by different users to learn new computing tasks.

Typically, cloud computing resources are provided to a user on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g. an amount of storage space consumed by a user or a number of virtualized systems instantiated by the user). A user can access any of the resources that reside in the cloud at any time, and from anywhere across the Internet. In context of the present invention, a user may access applications or related data available in the cloud. For example, the user assistance application212could execute on a computing system in the cloud. In such a case, the assistance application212may translate platform specific action information to action information understood by a target device at a storage location in the cloud. Similarly, the assistance application212could translate the language of instructions (auditory or visual) to a preferred language of a target user. Doing so allows a user and instances of the assistance application212to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).