Patent Document ID: 9916328
Application ID: 14451393
Patent Status: 1

Claim One:
1. A system comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the system to: receive a first image of a screen captured on a mobile device, the screen including content generated by a mobile application currently executing on the mobile device, identify recognized items in the first image by performing recognition on the image, provide the recognized items along with recognized items from at least one previous screen captured on the mobile device within a window of captured screen images to a machine learning model trained to predict an action element for at least one recognized item of the recognized items in the first image based on the provided recognized items and based on past actions performed by users, determine an assistance window for the action element, generate annotation data that includes the assistance window, and provide the annotation data for display with a current screen generated by the mobile application currently executing on the mobile device.