System and method for interacting in layers in channels over the display of a resource by another application

Methods, apparatus and software allowing users to exchange and display information regarding the resources each of them is viewing. This information, which may comprise one or more of chat threads, stickers and drawings, is displayed at user display locations in an overlay to underlying electronic information resources.

A portion of the disclosure of this patent document contains material which is subject to (copyright or mask work) protection. The (copyright or mask work) owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the patent and trademark office patent file or records, but otherwise reserves all (copyright or mask work) rights whatsoever.

FIELD OF THE INVENTION

The invention relates generally to graphical user interfaces enabling users to automate processing of and action over electronic information resources generated by underlying software applications and graphically showing the results in a visual overlay.

BACKGROUND OF THE INVENTION

Users employ software applications to search for, view information resources. These individual software applications have varying capabilities for automating user actions to finding, filtering, analyzing and updating the information resources.

Without automated support for processing results displayed by the underlying software application, achieving desired objectives can be very time consuming and mentally taxing for users, for example requiring them to copy reference identifiers, mentally memorize and compare displayed results to satisfy a set of conditions they hold in their mind. Furthermore, when undertaking numerous and/or complex tasks users experience cognitive stress which can be relieved by having a system that a) enables them to organize activities and tasks within distinct channels and b) remembers for them the display location context of marks that they place over that information. The benefits of these capabilities include that, rather than having to bookmark web pages, take screenshots, or maintain a separate document with links and comments, these activities are captured as part of the task process with no special user effort.

Furthermore, users need automated support regardless of the type of computing device or several devices that they may be using, where each may have different screen sizes and orientations.

BRIEF SUMMARY OF THE INVENTION

According to an embodiment of the present invention, a single user's activities, which may be one or more of:

examining a plurality of electronic information resources;

attaching a chat thread to a viewing position within an electronic information resource;

placing a sticker at a viewing position within an electronic information resource;

attaching a user-generated drawing to a viewing position within an electronic information resource;

injecting user-generated content such as photo images as new electronic information resources;

or data resulting from such activities are automatically associated as annotations overlaid at their designated display location in a channel which may be determined by the user.

According to an embodiment of the present invention, a user may switch from one channel to another and send one or more annotations to a different channel from a currently active channel.

According to an embodiment of the present invention, a plurality of users may participate in a channel to create one or more annotations that are shared between them within said channel.

According to an embodiment of the present invention, channels may be established that give some users restricted access, such as view only permissions.

According to an embodiment of the present invention, the system provides access to the annotations for the purposes of creating analyses of individual user activities, group user activities, activities within a channel, activities across a group of channels, real-time trending information by activity type and electronic information resource or combinations thereof.

DETAILED DESCRIPTION OF THE INVENTION

This is a continuation and priority is claimed back to U.S. patent application Ser. No. 13/938,122 SYSTEM AND METHOD FOR EXCHANGING AND DISPLAYING RESOURCE VIEWING POSITION AND RELATED INFORMATION filed Jul. 9, 2013, U.S. provisional patent application No. 62/196,856 SYSTEM AND METHOD FOR GENERATING INTERACTIVE LAYERS OVER THE DISPLAY OF A RESOURCE BY ANOTHER APPLICATION filed Jul. 24, 2015 and U.S. patent application Ser. No. 15/215,140 SYSTEM AND METHOD FOR GENERATING INTERACTIVE LAYERS OVER THE DISPLAY OF A RESOURCE BY ANOTHER APPLICATION filed Jul. 20, 2016 which are each hereby incorporated by reference in their entirety herein.

According to an embodiment of the present invention and as illustrated atFIG. 1, a user using a computing device101comprises a processor, program memory and data storage memory together with input/output hardware that connects the computing device to peripheral devices and a communications network106. Peripheral devices to the computing device may comprise one or more of a user display104, keyboard, pointing mechanism and other peripherals may include voice input, gesture recognition and motion detection devices. The user operates retrieval and display software102loaded onto the computing device101that initiates information requests and receives responses over a communications network106to display retrieved information103on the a user display104. Under this embodiment, the user loads user layer software105to process retrieved information108according to the specifications contained in one or more sets of user layer configuration data107to generate one or more user layer displays302.

In a preferred embodiment, a user may control the behavior of the user layer software105through a main task interface for which a control flow chart is illustrated atFIG. 2. A user may decide whether or not to use user layer software105and if they do they may create and edit one or more sets of user layer configuration data107and control which of said sets of user layer configuration data are active for the generation of user layer displays302.

In a preferred embodiment, each user layer's behavior, resulting from the application of user layer configuration data107to retrieved information108by user layer software105, is configurable using parameters and instructions supplied by a user501that may comprise comparison tests against one or more of keywords, strings of words or phrases, logic sequences, language expressions, paradigms, object recognition and target outcomes as further illustrated atFIG. 7.

In the preferred embodiment user layer software105processes one or more sets of user layer configuration data107to compute one or more graphical overlays303for display in context with associated retrieved information108as rendered by retrieval and display software102on a user display104within a user display window301and this is illustrated inFIG. 3.

In another embodiment a graphical overlay303may be separately displayable, even to the extent of being shown in a separate user display window301than retrieved information108. In combination, whether shown in the same window or not, rendered retrieved information108plus one or more graphical overlays303form a user layer display302.

A preferred embodiment is illustrated inFIG. 4which is a flow diagram of core functions performed by the user layer software105on an active set of user layer configuration data107. Where user layer configuration data107contains parameters and instructions that require data in addition to retrieved information103the user layer software105executes actions to access that additional data which may reside in a user layer history110or, as inFIG. 5, on a storage device503attached to a resource server accessed via a communications network106.

FIG. 5shows a preferred embodiment where a user's501computing device101is connected to a communications network106via which a user may cause the retrieval and display software102running on the device to make requests to information resource servers502which in turn accesses information storage devices503to access information resources504which are sent as a response to the computing device101over the communications network where they become retrieved information108for rendering by the retrieval and display software102on the user display104and for any processing by the user layer software105.

In the preferred embodiment, user layer configuration data107and user layer history110information may also be stored on a layer server601as illustrated inFIG. 6. A user501may thus access their user layer configuration data107and said histories thereof using any computing device101from which they are able to sign in to said layer server.

In a preferred embodiment as illustrated atFIG. 7, user layer software105provides a user501the ability to search and optionally edit one or more of user layer configuration data107and user layer history110data.

In a preferred embodiment, user layer software105allows for the naming of each set of user layer configuration data107, illustrated inFIG. 7, for easy reference and for sets of user layer configuration data107to be grouped together to form a layer group and for each group to be named and activated together

In the preferred embodiment, through layer precedence information stored within user layer configuration data107illustrated inFIG. 7, a user501may combine two or more sets of user layer configuration data107through cross-reference information that defines one or more connections between such sets of user layer configuration data107including the precedence of their processing and the condition activating said cross-reference; for example, one said set of user layer configuration data107may highlight duplicate items within the retrieved information108, one said set may search for lowest price information for non-duplicate items by accessing additional retrieved information108and another said set may compare the results with prior results stored in the user layer history110, all resulting a graphical overlay303annotating unique items with the current lowest price for each and prior lowest prices.

In the preferred embodiment, a set of user layer configuration data801to be processed is loaded802into data storage memory108for user layer configuration data107in the user's computing device101where it is processed by user layer software105against retrieved information103. A graphical overlay303, if any results, is output803by user layer software105as part of a user layer display302in context with the rendered retrieved information103.FIG. 8provides a schematic representation of the above steps.

In the preferred embodiment, use layer software105stores a user layer history110that retains retrieved information108and results of its processing said retrieved information108according to user layer configuration data107. An illustration of including user layer history110to produce a graphical overlay303is contained inFIG. 9, which is identical toFIG. 8except that the user layer configuration data107has been edited to trigger a “Lower price” comparison rather than “Similar”. For example, if a user is conducting a search of the Internet using retrieval and display software102and wishes to know which elements or group of elements in resulting retrieved information108were already returned by their prior searches, such conditions would be specified by the user as part of user layer configuration data108which user layer software105would process to detect such conditions and generate a graphical overlay303to assist the user in distinguishing new results from prior duplicates.

In a preferred embodiment, sets of user configuration layer data107are configurable by a user to automatically cause actions. For example, an automated process for finding matching data elements across a range of information resources504may be organized as follows: user layer configuration data107causes the user layer software105to first search a web page stored as retrieved information108for specific tagged elements by invoking the standard javascript function Element.getElementsByTagName( ) to return a list of all elements in said retrieved information108matching a tag name specified in user layer configuration data107. Second, from the locations of other information resources504supplied as part if user layer configuration data107, said user layer software105initiates retrieval and a search of said additional retrieved information108from said other information resources504to identify elements or groups of elements that match said tag name and generate a graphical overlay303highlighting elements for which matches were found and any ancillary information specified in the user layer configuration data107such as price and color. The result of such an operation is illustrated atFIG. 10, in which the screen shot shows a graphical overlay303annotating similar results from said other information resources504within a user layer display302.

In the preferred embodiment, methods of annotation in graphical overlays303are specified in user layer configuration data107as illustrated atFIG. 7. By employing a variety of such methods, a user may define the color, style, positioning and other parameters of an annotation to distinguish between them and suit their preference either through public domain javascript methods such as window.alert (“annotation text”); to create a temporary pop-up window and rect(x, y, widthinpixels, heightinpixels); to draw a rectangle on top or proprietary or patented annotation methods. In this way, user configuration data107is used by user layer software105to display different distinguishing marks, style or modes of annotation in order to differentiate between results in a graphical overlay303.

The preferred embodiment includes logic within the user layer software105to identify which subset of retrieved information103information has been visibly displayed within the display window301rendered by the retrieval and display software102. The preferred embodiment accomplishes this by capturing a user's viewing position by means of positional parameters using one or more methods anticipated in claims 2 through 8 of patent application Ser. No. 13/938,122 filed Jul. 9, 2013 and storing said positional parameters as part of user layer history110data. User layer configuration data107may, therefore, additionally determine whether the user layer software105acts or acted on all retrieved information108or only retrieved information108that was visible to the user using viewing position information collected within user layer history data110as illustrated inFIG. 7.

The preferred embodiment of user layer software105may further assist the user501by allowing them to define how multiple sets of user layer configuration data107are applied to retrieved information108. This embodiment allows a user501to specify, for example, sets of user layer configuration data107that should be applied on retrieved information108by user layer software105in series like successive filters, whether they are applied in parallel and whether iteration is applied. For example, by applying sets of user layer configuration data107in series user layer software105will produce a user display104annotating items that meet the conditions in all sets of user layer configuration data107whereas, by applying sets of user layer configuration data107in parallel, user layer software105will produce a user display104annotating items that meet the conditions defined within said sets of user layer configuration data107and when applying sets of user layer configuration data107iteratively, user layer software105will produce a user display104that is updated when underlying conditions change. This preferred embodiment of combining application of multiple sets of user configuration data107also solves for the user501problem of identifying similar but unique elements in a single graphical overlay303within a user layer display302. A example of combining user layers to produce a graphical overlay303showing how many comparable items were found in a reference database is illustrated at the block diagram atFIG. 11and an example block diagram of multiple user layers that together seek comparable items, seek pricing for them and displaying changes against historical results in a graphical overlay303, is shown atFIG. 12.

In a preferred embodiment, user layer software has access to a user's viewing history as an information resource504and user layer history101from which is produced a graphical overlay303that annotates a portion or portions of previously retrieved information108already seen by the user in the manner exemplified inFIG. 13.

In a preferred embodiment, user layer configuration data107is configurable by the a user to access one or more reference data sources as an information resource504such that elements within retrieved information108that also occur in said reference data are highlighted within a graphical overlay303. Such a configuration is illustrated atFIG. 14.

In a preferred embodiment, user layer software105allows a user501to suppress a graphical overlay303, thereby permitting them to see underlying rendered retrieved information108uncluttered by graphical overlay(s)303.

The preferred embodiment of user layer software105assists users501in organizing their work by adding category meta-data, see example atFIG. 7, to user layer configuration data107.

The preferred embodiment of user layer software105provides users501the ability to share at least one of user layer configuration data107, graphical overlay303, user layer display302and user layer history110with other users by stipulating so as illustrated inFIG. 7. Thus, an investment by one user in creating a user layer may be transferred for reuse to another user with the potential for monetary exchange in return for such transfer or reuse.

According to another embodiment of the invention, an apparatus for employing user layer software includes a display, an input/output hardware unit and a program memory coupled to a processor. Input/output hardware unit is capable of exchanging data with a network and data associated electronic resources accessible over a network. Program memory stores program instructions and a processor executes program instructions to (i) produce output from one or more software applications, (ii) execute user layer software for any user layer configuration data that has been enabled for those software applications, and (iii) display a graphical overlay derived from the output of user layer software placed in context with the output from said software applications in accordance with user layer configuration data.

According to another embodiment of the invention, a system for employing user layer software includes a database, an input/output hardware unit, and a program memory coupled to a processor. Input/output hardware unit is capable of exchanging data with a network. Program memory stores program instructions and the a processor is capable of executing program instructions to: (i) store and retrieve user layer configuration data from the database (ii) store and retrieve from the database results from the user layer software having been executed using user layer configuration data iii) initiate exchange with other users of user layer configuration data, iv) initiate exchange with another user of results from user layer software having been executed using their user layer configuration data.

According to another embodiment of the invention, a computer program product includes computer program logic stored in a tangible medium, wherein computer program logic executes program instructions to (i) execute user layer software for any user layer configuration data that has been enabled for a software application, and (iii) display a graphical overlay derived from the output of user layer software over the output from said software applications in accordance with user layer configuration data.

According to another embodiment of the invention, a computer program product includes computer program logic stored in a tangible medium, wherein computer program logic includes logic for causing a computer program to: (i) store and retrieve user layer configuration data from the database (ii) store and retrieve from the database the results from user layer software having been executed using user layer configuration data iii) initiate exchange with other users of user layer configuration data, iv) initiate exchange with another user of the results from user layer software having been executed using their user layer configuration data.

According to an embodiment of the present invention and as illustrated in the figures from which the present invention is a continuation, a user using a computing device comprises a processor, program memory and data storage memory together with input/output hardware that connects a computing device to peripheral devices and a communications network. Peripheral devices to the a computing device may comprise one or more of a user display, keyboard, pointing mechanism and other peripherals may include voice input, gesture recognition and motion detection devices. A user operates retrieval and display software loaded onto the a computing device that initiates information requests and receives responses over a communications network to display retrieved information on the a user display. Under this embodiment, a user105loads user layer software to process retrieved information in relation to the user's display location within the retrieved information to display overlay information.

In a preferred embodiment, a user501may organize their electronic information resource1604viewing activities into channels1501or expeditions as illustrated inFIG. 15, each of which is represented by a channel thumbnail1502comprising one or more of a channel image1503, a channel name1504, last channel activity information1505and channel user role information1506. In a preferred embodiment, channels thumbnails1502may be in channel categories1507and may be searched through a channel search box1508to allow for the case, for example, of finding all channels with a particular user as a member.

In a preferred embodiment, once a user501has entered a channel they will be presented a channel screen as illustrated atFIG. 16comprising one or more of:a channel header1601containing one or more of a channel name1504, channel user role information1506, channel category1507, channel member information1602which allows access to details of members of the a channel in the case there is a plurality of users with access to the a channel, a channel activity indicator1603which allows access to the activity in the a channel which may be spread over more than one electronic information resource and in some embodiments additional information including the number of users in a channel, the number of resources being viewed within a channel or expedition or other information related to activity in the a channel;a resource display area1604showing an electronic information resource currently being viewed and optionally containing a sharing control element1605allowing a current user to select a specific display location for one or more of a new chat thread, sticker, drawing including a user generated drawing or insertion of other user generated content such as a photo image;a resource browsing control area1606that permits a user to search for and select an electronic information resource to be showing in the a resource display area1604;an annotation notification area1607providing information about activities not always visible within a current electronic information resource display area1604for the a channel1501currently in focus.

In a preferred embodiment illustrated atFIG. 17, activating channel member information1602which allows access to details of members of the a channel in the form of a channel member list1701.

In a preferred embodiment illustrated atFIG. 18, activating a channel activity indicator1603allows access to activity and any resulting annotations in the a channel which may be spread over more than one electronic information resource in the form of a channel activity list1801.

In a preferred embodiment illustrated atFIG. 19, activating a sharing control element1605permits a user to create one or more of:

a chat thread attached to a display location which may be displayed in an overlay to a current electronic information resource1604as either a compact or closed chat thread indicator1701or an open chat thread1702;

a sticker attached to a display location1703in an overlay to a current electronic information resource1604.

In a preferred embodiment illustrated atFIG. 20, activating the a sharing control element1605permits a user501to create a drawing2001as an overlay to an electronic information resource1604. In the case where a plurality of users is interacting using the present invention those users can, in some embodiments providing they have access to the channel, separately and contemporaneously create drawings in the same channel as an overlay to a same electronic information resource as additionally exemplified as drawing from user A at2001and drawing from user B at2902.

In a preferred embodiment also illustrated atFIG. 20. a current electronic information resource1604may also include user generated content such as photo images exemplified here at2003and with simultaneously drawn lines from a plurality of users501exemplified as drawing from user A at2004and drawing from user B at2005.

In a preferred embodiment a user501interacts with a channel user interface displaying channels1501using a method illustrated inFIG. 21which allows users501to select from the at least one channel using its thumbnail1502. Referring toFIG. 21, a user may press a menu button in2101resulting in a user having access to a profile and settings menu or menus. There, the a user may edit their profile and settings by selecting from a submenu or entering information. Also referring toFIG. 21, a user may further select an annotation notification2102, which allows a user to view and act on annotations by other users.FIG. 21also illustrates that by selecting the a channel name2013a user is able to view channel detail information including which other users are members of the a channel. InFIG. 21, selecting an unread bell2014provides a user with a list of all annotation notifications that they have not yet read and a channel search function2105enables a user to select a set of channels fitting a set of search characteristics they enter. Subsequent processes may include those depicted inFIGS. 22 through 25.

In a preferred embodiment electronic information resources1604are displayed in an Internet browser view integrated with overlaid annotations e.g.1901,1902and1903and any annotation notifications using the method illustrated atFIG. 22. The approach illustrated allows each user501to browse any electronic information resource1604independently of the activities of any other user501in the a current channel1501or any other channel1501.

In a preferred embodiment also as illustrated inFIG. 22, annotations e.g.1901,1902and1903are initiated by users501using a share button1605indicator which is dragged and dropped to a display location within a current electronic information resource1604viewing method as indicated in a drag and drop operation at2201. InFIG. 22are also shown methods that are added to standard browser operations2204which include accessing annotation notifications2202through annotation notification elements1607and the ability to modify how overlaid annotations e.g.1901,1902and1903are displayed using selector2203.FIG. 22also illustrates a method of accessing information relevant to a channel1501through a channel header1601by selecting action elements2205.

In a preferred embodiment a user501, which includes a joining user, can initiate new overlaid annotations using the method illustrated atFIG. 23which also permits an overlaid annotation to be used to start new channels1501.FIG. 23illustrates a method that can be used following selection of the a drag and drop operation2201such that a user may choose one or more options such as sending an overlaid annotation to a different or new channel2301, select an annotation type such as a doodle2302, sticker2303or text2304, repositioning a draft overlaid annotation2305to a different viewing position and communicating a finalized annotation2306through a chosen channel1501. An annotation e.g.1901,1902and1903information is communicated to other users through a communications network to which users are connected as illustrated inFIG. 5andFIG. 6. Annotations may be stored by all users501or stored for reference on one or more servers501.

In a preferred embodiment, a user501can manage their interactions with other users including which other users501have access to channels1501they created or are leading through either a main menu, a channel thumbnail1502or a channel header1601through a member detail process as illustrated atFIG. 24.FIG. 24shows a method in which members may be added or modified2401whether in a user's contact list or not, in which the a current user may select a specific member2402to see more detail and optionally jump to that member's viewing position2403.FIG. 24also illustrates the ability to block or suppress2404overlay annotations and associated annotation notifications.

In a preferred embodiment users501that are members of a channel1501may receive notifications of annotations in channels1501they are members of and can view and respond to those annotations using the method illustrated inFIG. 25. As illustrated inFIG. 25, a user may access annotation notifications in the form of a channel list2501or an unread notification list2502and select to jump2503to the a browser view of that annotation e.g.1901,1902and1903or to make a quick reply2504directly from an annotation notification area concerned.

According to some embodiments of the invention, a computer of mobile device includes a display, a processor, memory and input/output devices, such as a touch screen, buttons, keys, microphone, and speakers and a network interface for coupling the computer electrically, wirelessly or optically to at least one network, such as Wi-Fi network, a LAN, WAN or the Internet. The processor is coupled to the display, memory, input/output devices and the Internet and executes program instructions stored in the memory to perform various functions described herein.

According to some embodiments, the display includes a plurality of channels and other information as shown inFIGS. 15-20. The memory may store an application or app which allows users to login or logout, which information may be stored on a remote server. Channels shown on the display may be created when one or more users use the application to access content, for example on the Internet or another network, and the number of users in the channel can grow as users invite other users of the app or other users join a channel, who are permitted to do so by the app or by friendship status among one or more users. Data associated with a user's browsing within the app and/or channel within the app is stored to facilitate the collection and sharing of user activities described herein.

According to one embodiment, a user may sign in, view channels he or she is permitted to join, join channels, receive notifications of the activities of other users, and select notifications and be transported to electronic resources that are annotated by other users in the channel based on various user activities as described herein. Users may also access content while affiliated with one or more channels and perform activities that are pushed out to other users associated with or within at least one channel in the form of notifications or stored data on the user's browsing activity and annotations. The app itself may also provide statistics relating to the channels, update icons corresponding to channels based on user activity and provide other information to facilitate enabling users to engage and annotate content and perform activities around content in a channel.

While particular embodiments are disclosed herein, it will be understood by one having ordinary skill in the art that changes may be made to those embodiments without departing from the spirit and scope of the invention.