GENERATING INTERACTIVE SCREENSHOTS USING A FEDERATED NETWORK

Aspects of the disclosure provide for mechanisms for generating interactive screenshot based on a screenshot. A method of the disclosure includes retrieving a screenshot by a first computing device of a plurality of computing devices, identifying a subset of the plurality of computing devices, transmitting a unique packet of data associated with the screenshot for each of the subset of the plurality of computing devices, obtaining an intermediate result used to generate an interactive screenshot based on the unique packet of data, and generating the interactive screenshot based on the intermediate results of each of the subset of the plurality of computing devices.

TECHNICAL FIELD

Embodiments of the disclosure relate generally to memory sub-systems, and more specifically, relate to generating interactive screenshots using a federated network of computing devices.

BACKGROUND

Content sharing (e.g., articles, restaurants, shows, songs, videos, social media posts) is typically performed through Uniform Resource Locator (URL) sharing or screenshot sharing of the content.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed to a federated network of computing devices used to generate interactive screenshots based on static screenshots. Screenshots can be static digital images that capture the display's contents (e.g., a computer device, mobile device, etc.). Screenshots are created by the operating system or software running on the computer or mobile device, for example. In general, screenshots can be utilized to share the contents of the display with other computer devices or mobile devices. Content sharing can be performed through a URL (e.g., link), which provides the receiver with a link to the content or the specific location of the web-based application where the content is located. However, with no image, the receiver has no immediate reference to the purpose or intention behind the URL. In some embodiments, the URL may be shared with an auto-populated snapshot of the content during sharing of the URL. However, in these instances, the snapshot of the content is directed to the content as a whole rather than a specific portion that may have been of interest to the receiver.

Content sharing can be done by sharing a screenshot that provides an image reflecting the content, however the receiver cannot immediately access the content. While the intent and purpose of the content can be actualized from the screenshot by the receiver, the receiver is left without means to interact with the content, manually search the Internet or other applications to interact with the content, or forgoing access to the content. Further, in some embodiments, the sender of the screenshot or user who screenshotted the content may lose track of a location (e.g., the URL or application) that corresponds to the content, in which case the user, receiver, and/or sender would have to scrape the Internet and other applications manually to find the content to interact with it directly.

Aspects of the present disclosure address the above and other deficiencies by having a federated network of computing devices (and, in some instances, servers) combine a static screenshot of content with a link and/or an applet to generate an interactive screenshot (or experience). One or more computing devices of the federated network (further referred to as federated network) identify an application within the static screenshot. Based on the identified application, the federated network extracts information from the static screenshot. Based on the identified application and extracted information from the static screenshot, the federated network performs a query search to generate a URL corresponding to the static screenshot as a whole and/or each portion (or sub-portion) of the static screenshot. Based on the generated URL, the federated network updates the static screenshot as a whole and/or the relevant portions of the static screenshot with the URL (or data derived from the URL) to generate an interactive screenshot.

Advantages of the present disclosure include, but are not limited to, reducing the reliance on a server to generate interactive screenshots, thereby providing a decentralized approach to generating interactive screenshots. Various aspects of the above-referenced methods and systems are described in detail below by examples, rather than by limitations.

FIG.1illustrates a computing device110in which implementation of the disclosure may operate. Computing device110may be a desktop computer, a laptop computer, a smartphone, a tablet computer, a server, a scanner, or any suitable computing device capable of performing the techniques described herein. Computing device110includes a screenshot conversion module120, a storage130, a machine learning component140, and a network interface card (NIC)150. In some implementations, computing device110can be (and/or include) one or more computing devices700ofFIG.7.

The screenshot conversion module120may generate an interactive screenshot from screenshot135stored in storage130. The screenshot conversion module120uses image processing and/or machine learning to identify an application within screenshot135. Contents are extracted from within the identified application within screenshot135. The extracted contents are used to generate the interactive screenshot.

Screenshot135may be a visual representation of an application, an operating system, a document containing images, text, and/or URL links. The screenshot conversion module120retrieves screenshot135from a storage device. The storage device may be internal storage130or an external storage device (e.g., cloud). Screenshot135may include varying degrees of metadata. In some instances, the metadata of screenshot135may not be sufficient to determine which application was captured in the screenshot screenshotted.

In some embodiments, the screenshot conversion module120may identify an application in screenshot135by identifying structural elements (e.g., application markers) within screenshot135. The screenshot conversion module120may use machine learning (e.g., convolution neural network (CNN), artificial neural network (ANN), deep neural network (DNN), etc.) to identify structural elements within a screenshot. Structural elements may include icons, buttons, titles, logos, or other suitable GUI elements. The application may be a social media application, a music application, a web page, a web browser, a messaging application, or any other third-party application.

The screenshot conversion module120may extract each structural element (e.g., application markers) of the identified application and content element (e.g., content marker) from within the identified application within screenshot135. Content elements may include text, images, URLs, or any other suitable information associated with the application's contents. For example, for a social media application, the extracted content elements can include images, user name, comments, number of comments, duration of a video post, text, date, and time of the post.

The screenshot conversion module120may perform a query search for each extracted element and/or structural element to obtain an interactive element. In particular, the screenshot conversion module120may query, via local databases, external databases, and/or other searching techniques, to obtain links and/or applets for each extracted element and/or structural element. Searching techniques can include web-crawling (or web-scraping), an application programming interface (API) of the identified application, a software development kit (SDK) of the identified application, semantic text matching, reverse image searching, etc. Links may be an application programming interface (API), a digital object identifier (DOI), a uniform resource identifier (URI), a uniform resource locators (URL), etc. Applets may be small applications (or programs) with various functionality that resembles the functionality of an extracted element and/or a structural element. For example, an applet may be developed and stored in the local and/or external database to mimic the functionality of the extracted element and/or structural element or the applet may be a third-party applet associated with the identified application that provides the functionality to the extracted element and/or structural element.

In some embodiments, additional information from the computing device110may be used by the screenshot conversion module120to assist in identifying the application within the screenshot and/or performing the query search for extracted elements from the identified application. The additional information may include sensor data, application programming interface (API), location services, and data from internal memory, including installed applications, cache files, etc. For example, the screenshot conversion module120may access the internal memory to obtain a list of installed applications and determine whether the structural elements correspond to one of the installed applications. In another example, the screenshot conversion module120may access location services to identify an area that limits the query search of extracted elements. The area may be a radius area indicating a likely area in which the screenshot was taken. In another example, the screenshot conversion module120may access cache files associated with each installed application to limit the query search of extracted elements.

The screenshot conversion module120may update each element of screenshot135with the interactive element used to redirect, via a link, a user to a source of the content associated with the an element (e.g., extracted element and/or a structural element) or provide functionality, via an applet, corresponding to the content associated with the element (e.g., extracted element and/or a structural element) of screenshot135. In some embodiments, providing functionality via the applet includes providing a graphical user interface as the interactive element which when interacted with reproduces the functionality of the element.

FIG.2illustrates a federated network200that includes server230and computing devices210A-E, similar to the computing device110ofFIG.1. Server230and each of the computing devices210A-E is interconnected to one another via a peer-to-peer (P2P) network. In some instances, the server and each of the computing devices210A-E may be interconnected to one another via a network280. Network280may be a public network (e.g., Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof.

Server230and each of the computing devices210A-E, similar to computing device110ofFIG.1, includes a screenshot conversion module120. One of the computing devices210A-E (e.g., originating device) may retrieve a screenshot to generate an interactive screenshot. The originating computing device (e.g., computing device210A) may rely on the federated network200to generate the interactive screenshot.

In some embodiments, the originating computing device may provide the screenshot to the screenshot conversion module120of server230to generate the interactive screenshot. Accordingly, Server230may identify the application within the screenshot, extract elements from within the identified application, perform a query search on the extracted elements, and generate an interactive screenshot based on the query search results.

In another embodiment, the screenshot conversion module120of the originating computing device may identify the application within the screenshot and extract elements from within the identified application. The originating computing device may provide the extracted elements to the screenshot conversion module120of server230configured to perform query searches. Accordingly, Server230may return a link or applet for each extracted element from the originating computing device. The screenshot conversion module120of the originating computing device receives the link or applet and generates an interactive screenshot based on the received links or applets.

Regardless of the approach, server230may identify the amount of computational power necessary to fulfill the request of the originating computing device. Server230may request computing resources from a subset of the computing devices210A-E. In some embodiments, the server may request computing resources from all of the computing devices210A-E. Each of the requested computing devices may provide their available computing resources (e.g., computing resource215B of computing device210B) to server230.

In some embodiments, each computing device that provided computing resources may be compensated. Compensation may be in digital currency (e.g., cryptocurrency, tokens), fiat currency, complimentary use of the screenshot conversion module, etc.

Server230, using the received computing resources of the requested computing devices, may proceed with fulfilling the request of the originating computing device. Depending on the embodiment, server230may request computing resources from the computing devices210A-E in anticipation of a request from the computing devices210A-E.

Depending on the embodiment, server230may be a mobile web server instantiated within the originating computing device. Accordingly, the originating computing device or server230may identify an amount of computational power necessary to fulfill the demands of the originating computing device. The originating computing device and/or server230may request and receive computing resources from the remaining computing devices (e.g., computing devices210B-E). Server230, using the received computing resources of the requested computing devices, may fulfill the demands of the originating computing device instead of the originating computing device.

FIG.3illustrates a federated network300that includes computing devices310A-E, similar to the computing device110ofFIG.1. Each of the computing devices310A-E is interconnected to one another via a peer-to-peer (P2P) network. In some instances, each of the computing devices310A-E may be interconnected to one another via a network380. Network380may be a public network (e.g., Internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof.

Each of the computing devices310A-E, similar to computing device110ofFIG.1, includes a screenshot conversion module120. One of the computing devices310A-E (e.g., originating device), may retrieve a screenshot to generate an interactive screenshot. The originating computing device (e.g., computing device210A) may rely on the federated network300to generate the interactive screenshot.

The originating computing device may provide the screenshot conversion module120of the remaining computing devices (e.g., computing devices210B-F) a packet of data. In some embodiments, only a subset of the remaining computing devices may be designated to receive the packet of data to process. The packet of data may be a screenshot (or a portion of the screenshot) retrieved by the originating computing device and/or an intermediate result of a screenshot conversion module. The intermediate result of the screenshot conversion module may include structural elements of the screenshot, content elements of the screenshot, link and/or applet, query search results, etc. The intermediate results of the screenshot conversion module may be any information produced by one or more steps of the screenshot conversion module.

Depending on the embodiment, the originating computing device may instruct each of the remaining computing devices to process the packet of data similarly. In an example, each of the remaining computing devices, using the received packet of data (e.g., a screenshot), performs all the steps necessary to generate an interactive screenshot. As noted above, the steps to generate the interactive screenshot may include identifying the application within the screenshot, extracting elements from the identified application, obtaining links and/or applets for extracted elements, and generating the interactive screenshot using the obtained links and/or applets. In another example, each of the remaining computing devices, using the received packet of data (e.g., extracted elements), performs a specific step (e.g., obtaining links and/or applets for the extracted elements).

In an embodiment, the remaining computing devices may reach a consensus as to which result of the remaining computing devices will be provided to the originating computing device. The remaining computing devices may reach a consensus using any suitable consensus mechanism that allows a network (e.g., federated network200) to agree on the state of the results. Some examples of consensus mechanisms may be proof of stake, delegated proof of stake, proof of work, proof of elapsed time, proof of activity, etc.

In another embodiment, the originating computing device may receive the results (e.g., the interactive screenshot, the obtained links and/or applets, etc.) from each of the remaining computing devices and identify the most accurate result. In some embodiments, the originating computing device may identify the most accurate result based on a confidence score (or trust score) assigned to each of the remaining computing devices. Accordingly, the result associated with the computing device among the remaining computing device with the highest confidence score will be accepted as the most accurate result (or true result). The confidence score of each of the remaining computing devices may be assigned and updated based on a user's input regarding the accuracy of the results. For example, the user of the originating device may identify whether one or more obtained links redirected the user to the appropriate location within the identified application, whether the identified application was correct, etc. Other methods of assessing the accuracy of the remaining computing devices to process the packet of data are contemplated. Depending on the embodiment, the most accurate result may be manually selected.

Depending on the embodiment, the originating computing device instructs each remaining computing device to process a unique packet of data. In an example, each of the remaining computing devices, using the received packet of data (e.g., a different portion of the screenshot), performs all the steps necessary to generate an interactive screenshot. In particular, the steps include identifying the application based on the portion of the screenshot, extracting elements from the portion of the identified application, obtaining links and/or applets for extracted elements, and generating a portion of the interactive screenshot using the obtained links and/or applets. The originating computing device may combine the individual portions of the interactive screenshot into a cohesive interactive screenshot or provide the individual portions of the interactive screenshot as a packet of data to another one of the remaining computing devices to combine.

In another example, each of the remaining computing devices receives a packet of data necessary to perform a specific step. In particular, a remaining computing device may receive structural elements to perform the step of identifying the application, another remaining computing device may receive the identified application to perform the step of extracting elements from the identified application, and so on.

Each result generated by a remaining computing device may be forwarded to the originating computing device to determine whether the originating computing device or another one of the remaining computing devices will perform the next step in generating the interactive screenshot. Based on the determination, the originating computing device will either perform the next step or send a packet of data necessary to perform the next step to another remaining device. The originating computing device continues until the interactive screenshot is generated. The generated interactive screenshot is then provided to the originating computing device. Depending on the computing device that generated the interactive screenshot, the generated interactive screenshot is then provided to the originating computing device.

Alternatively, each remaining computing device may forward their respective results to another one of the remaining computing devices to perform the next step in generating the interactive screenshot. The results are constantly forwarded to another one of the remaining computing devices until the interactive screenshot is generated. Depending on the computing device that generated the interactive screenshot, the generated interactive screenshot is then provided to the originating computing device.

Each computing device that was instructed to process a packet of data may be compensated. Compensation may be in digital currency (e.g., cryptocurrency, tokens), fiat currency, complimentary use of the screenshot conversion module, etc.

It should be noted that while different approaches with respect to the federated network200and300, any combination of approaches may be utilized to generate an interactive screenshot.

At operation410, the processing logic retrieves, by a first computing device of a plurality of computing devices, a screenshot. The screenshot may be stored in a local storage of the first computing device or an external storage device.

At operation420, the processing logic identifies a subset of the plurality of computing devices. As previously described, the subset of the plurality of computing devices may be the remaining computing devices or any combination of the plurality of computing devices.

At operation430, for each of the subset of the plurality of computing devices, the processing logic transmits a unique packet of data associated with the screenshot. The packet of data may be structural elements of the screenshot, an identified application within the screenshot, extracted elements from the identified application within the screenshot, or links and/or applets associated with the extracted element. Each of the subset of the plurality of computing devices gets a unique or different packet of data. In some embodiments, the packet of data may be a portion of the screenshot. Each of the subset of the plurality of computing devices gets a unique or different portion of the screenshot (packet of data).

At operation440, the processing logic obtains, based on the unique packet of data, an intermediate result used to generate an interactive screenshot. The intermediate result may be an identified application, extracted elements, links and/or applets, or interactive elements. The intermediate results are obtained by processing, using the screenshot conversion module, the unique packet of data and arriving at the intermediate result. Depending on the embodiment, if the packet of data is a portion of the screenshot, the intermediate result that is returned from the screenshot conversion module is the portion of the screenshot with interactive elements.

At operation450, the processing logic generates, based on the intermediate result, the interactive screenshot. The individual intermediate result (e.g., link and/or applet associated with the extracted element) is used to generate interactive elements and update the screenshot to the interactive screenshot. In other instances, the individual intermediate result (e.g., a portion of the screenshot with interactive elements) are combined with other individual intermediate results to generate the full interactive screenshot.

At operation510, the processing logic retrieves, by a first computing device of a plurality of computing devices, a screenshot. The screenshot may be stored in a local storage of the first computing device or an external storage device.

At operation520, the processing logic transmits, to a server, a packet of data associated with the screenshot. The packet of data may be structural elements of the screenshot, an identified application within the screenshot, extracted elements from the identified application within the screenshot, or a link and/or applet associated with the extracted element. In some embodiments, the packet of data may be a portion of the screenshot.

The server is based on computing resources allocated by a subset of the plurality of computing devices. As previously described, the server request computing resources from the subset of the computing devices. Each of the requested computing devices provides its available computing resources to the server. The server, using the received computing resources of the requested computing devices, fulfills the request of the first computing device. As previously described, the subset of the plurality of computing devices may be the remaining computing devices or any combination of the plurality of computing devices.

At operation530, the processing logic obtains, by the server, an intermediate result used to generate an interactive screenshot based on the packet of data. The intermediate result may be an identified application, extracted elements, links and/or applets, or interactive elements. The intermediate results are obtained by processing, using the screenshot conversion module, the unique packet of data and arriving at the intermediate result. Depending on the embodiment, if the packet of data is a portion of the screenshot, the intermediate result that is returned from the screenshot conversion module is the portion of the screenshot with interactive elements.

At operation540, the processing logic generates, based on the intermediate result, the interactive screenshot. The individual intermediate result (e.g., link and/or applet associated with the extracted element) is used to generate interactive elements and update the screenshot to the interactive screenshot. In other instances, the individual intermediate result (e.g., a portion of the screenshot with interactive elements) is combined with other individual intermediate results to generate the full interactive screenshot.

At operation610, the processing logic retrieves, by a first computing device of a plurality of computing devices, a screenshot. The screenshot may be stored in a local storage of the first computing device or an external storage device.

At operation620, the processing logic identifies a subset of the plurality of computing devices. As previously described, the subset of the plurality of computing devices may be the remaining computing devices or any combination of the plurality of computing devices.

At operation630, for each of the subset of the plurality of computing devices, the processing logic transmits a packet of data associated with the screenshot. The packet of data may be structural elements of the screenshot, an identified application within the screenshot, extracted elements from the identified application within the screenshot, or a link and/or applet associated with the extracted element. Each of the subset of the plurality of computing devices gets the same packet of data. In some embodiments, the packet of data may be a portion of the screenshot. Each of the subset of the plurality of computing devices gets the same portion of the screenshot (packet of data).

At operation640, for each of the subset of the plurality of computing devices, the processing logic obtains, based on the packet of data, an intermediate result used to generate an interactive screenshot. The intermediate result may be an identified application, extracted elements, links and/or applets, or interactive elements. The intermediate results are obtained by processing, using the screenshot conversion module, the unique packet of data and arriving at the intermediate result. Depending on the embodiment, if the packet of data is a portion of the screenshot, the intermediate result returned from the screenshot conversion module is the portion of the screenshot with interactive elements.

At operation650, the processing logic obtains, by the subset of computing devices, a consensus on an individual intermediate result from the subset of the plurality of computing devices. As previously described, consensus may be reached using any suitable consensus mechanism (e.g., proof of stake). At operation660, the processing logic returns, to the first computing device, the individual intermediate result.

In some embodiments, the processing logic generates, based on the individual intermediate result, the interactive screenshot. The individual intermediate result (e.g., link and/or applet associated with the extracted element) is used to generate interactive elements and update the screenshot to the interactive screenshot. In other instances, the individual intermediate result (e.g., a portion of the screenshot with interactive elements) are combined with other individual intermediate results to generate the full interactive screenshot.

FIG.7depicts an example computer system700which can perform any one or more of the methods described herein. The computer system may be connected (e.g., networked) to other computer systems in a LAN, an intranet, an extranet, or the Internet. The computer system may operate in the capacity of a server in a client-server network environment. The computer system may be a personal computer (PC), a tablet computer, a set-top box (STB), a Personal Digital Assistant (PDA), a mobile phone, a camera, a video camera, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single computer system is illustrated, the term “computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

The exemplary computer system700includes a processing device702, a main memory704(e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory706(e.g., flash memory, static random access memory (SRAM)), and a data storage device716, which communicate with each other via a bus708.

The computer system700may further include a network interface device722. The computer system700also may include a video display unit710(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device712(e.g., a keyboard), a cursor control device714(e.g., a mouse), and a signal generation device720(e.g., a speaker). In one illustrative example, the video display unit710, the alphanumeric input device712, and the cursor control device714may be combined into a single component or device (e.g., an LCD touch screen).

The data storage device716may include a computer-readable storage medium724on which is stored the instructions726embodying any one or more of the methodologies or functions described herein. The instructions726may also reside, completely or at least partially, within the main memory704and/or within the processing device702during execution thereof by the computer system700, the main memory704and the processing device702also constituting computer-readable media. In some implementations, the instructions726may further be transmitted or received over a network via the network interface device722.