Patent ID: 12260004

The features and advantages of embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.

DETAILED DESCRIPTION

I. Introduction

The following detailed description discloses numerous embodiments. The scope of the present patent application is not limited to the disclosed embodiments, but also encompasses combinations of the disclosed embodiments, as well as modifications to the disclosed embodiments.

References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

In the discussion, unless otherwise stated, adjectives such as “substantially,” “approximately,” and “about” modifying a condition or relationship characteristic of a feature or features of an embodiment of the disclosure, are understood to mean that the condition or characteristic is defined to be within tolerances that are acceptable for operation of the embodiment for an application for which it is intended.

Furthermore, it should be understood that spatial descriptions (e.g., “above,” “below,” “up,” “left,” “right,” “down,” “top,” “bottom,” “vertical,” “horizontal,” etc.) used herein are for purposes of illustration only, and that practical implementations of the structures and drawings described herein can be spatially arranged in any orientation or manner. Additionally, the drawings may not be provided to scale, and orientations or organization of elements of the drawings may vary in embodiments.

Numerous exemplary embodiments are described as follows. It is noted that any section/subsection headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment may be included under any section/subsection. Furthermore, embodiments disclosed in any section/subsection may be combined with any other embodiments described in the same section/subsection and/or a different section/subsection in any manner.

II. Example Embodiments for Rendering Secured Content Items

Unbeknownst to a user, malware, such as a screen scraper or a keylogger, installed on the user's computing device may acquire sensitive or confidential information maintained by the computing device. For instance, when a user accesses a banking website or another sensitive web site, the web site may display confidential information, or the user may type confidential information into specific fields to acquire access. Such information may be captured by the malware and subsequently provided to a malicious entity, which utilizes such information for nefarious purposes.

The embodiments described herein are directed to rendering secured content items. For instance, a computing device may be utilized to view a content item that is to be displayed via a display device coupled thereto. However, rather than rendering the content item, the computing device generates and/or provides a graphical representation of the content item to a rendering device coupled between the computing device and the display device. The rendering device analyzes the graphical representation to determine characteristics of the graphical representation, characteristics of a display region of an application window in which the content item is to be rendered, and a network address at which the actual content item is located. The rendering device retrieves the content using the network address and renders the retrieved content over the display region of the application window in accordance with the characteristics determined for the graphical representation and the display region of the application window.

The embodiments described herein provide improvements in other technologies, namely data security. For instance, the techniques described herein advantageously prevent secured content items from being rendered and displayed when the rendering device is not coupled to the computing device, as the secure content item is not provided to the computing device for rendering thereby. Instead, the rendering of the secure content item is performed outside of the traditional graphic pipeline; in particular, the rendering is performed by the rendering device that is coupled between the computing device and the display device. This way, if malware, such as a screen scraper, is executing on the computing device, it would not have access to the secure content item, as the secure content item is not provided to and/or rendered by the computing device. Accordingly, the techniques described herein prevent access to secured content items, such as personal and/or confidential information, unless the rendering device is utilized.

In addition, the graphical representation of the secured content item may be a quick response (QR) code, which encodes the characteristics of the graphical representation, characteristics of the display region of the application window in which the content item is to be rendered, and the network address at which the actual content item is located. Such QR codes comprise a relatively small amount of data (e.g., up to 3 Kilobytes). Accordingly, a small amount of data is required to be transmitted to the rendering device and processed by the rendering device to retrieve and render the secured content item. Accordingly, the embodiments described herein also minimize the expenditure of computing resources (e.g., input/output operations, processor cycles, memory, power, etc.) for both the computing device that provides the graphical representation of the secured content item and the rendering device that analyzes the graphical representation and retrieves and renders the secured content item.

Systems and devices may be configured in various ways for rendering and displaying secure content items. For instance,FIG.1is a block diagram of a system100for rendering and displaying a secure content item via a rendering device externally coupled to a computing device in accordance with an example embodiment. As shown inFIG.1, system100comprises a computing device102, an external rendering device104, a display device106, and a server120. Computing device102is configured to execute one or more applications112. Application(s)112may be configured to access and/or open content items (e.g., content108) via a network116. Examples of content items include, but are not limited to, a document, a spreadsheet, a web page, an image, a video, and other forms of content (e.g., an application window and/or user interface elements in an embodiment in which a remote or virtual desktop is being provided via server120). Content item108may be hosted on server120(e.g., a web server, a data server, etc.) and/or provided via a web application executing on server120. Examples of application(s)112include, but are not limited, a browser application (e.g., Microsoft Edge®, published by Microsoft Corp. of Redmond, Washington, Mozilla Firefox®, published by Mozilla Corp. of Mountain View, California, Safari®, published by Apple Inc. of Cupertino, California, and Google® Chrome, published by Google Inc. of Mountain View, California), a word processing application (e.g., Microsoft Word® published by Microsoft® Corp.), a spreadsheet application (e.g., Microsoft Excel® published by Microsoft® Corp.), a presentation application (e.g., Microsoft PowerPoint® published by Microsoft® Corp.), etc. Network116may comprise one or more networks such as local area networks (LANs), wide area networks (WANs), enterprise networks, the Internet, etc., and may include one or more of wired and/or wireless portions.

Content (e.g., content item108) (either retrieved via network116or content local to computing device102) is rendered by a processor or graphical processing unit (GPU) of computing device102and typically provided to a display device (e.g., display device106) coupled to computing device104. Display device106and computing device102may be communicatively coupled via a communication interface, such as, but not limited to, an HDMI (High-Definition Multimedia Interface) interface, a USB (Universal Serial Bus) interface, a PCIe (Peripheral Component Interconnect Express) interface, a I2C (Inter-Integrated Circuit) interface, a DVI (Digital Video Interface), a DisplayPort interface, an embedded DisplayPort (eDP) interface, etc. Computing device102is configured to provide content for display by display device106via a video signal110, which is transmitted via such communication interface(s).

Computing device102may be any type of computing device or computing system, including, a personal computer, a laptop computer, a tablet device, a smart phone, a game console or gaming device, and/or the like that are configurable to output a video signal to a display device, such as display device106. Examples of display device106, include, but are not limited to, a monitor, a projector, a television, etc.

In certain scenarios, one may desire for content item108to be viewed by certain individuals and/or organizations. In such scenarios, content item108is made viewable via external rendering device104. As shown inFIG.1, external rendering device104is configured to be communicatively coupled to both computing device102and display device106. External rendering device104may act as an intermediary (or bridge) between computing device102and display device106. External rendering device104is external to computing device102(e.g., external rendering device104is not incorporated within computing device102). External rendering device104may also be external to display device106. Alternatively, one or more components of external rendering device104(as described herein) may be incorporated within display device106. In accordance with an embodiment, external rendering device104is a docking station to which computing device102(e.g., a laptop) may be attached. The docking station serves as a base station for a laptop, which comprises a plurality of input and/or output ports to which input and/or output peripheral devices may be attached. Examples of docking stations, include, but are not limited to, an expansion dock, a port replicator or hub, a breakout dock (or multi-port adaptor), etc. In accordance with another embodiment, external rendering device is a standalone (or custom) device that is configured to implement the techniques described herein.

When an application of application(s)112provide a request to server120for content item108, server120may instead provide code114. Application(s)112are configured to execute code114. Examples of code114include, but are not limited to JavaScript, TypeScript, etc. When code114is executed, a graphical representation118of content item108is rendered in an application window of the application of application(s)112that requested content item108. Graphical representation118may utilize an encoding to specify a network address (e.g., a uniform resource identifier (URI) or uniform resource locator (URL)) at which content item108may be retrieved. The encoding may also specify various characteristics of graphical representation118itself. For instance, the encoding may specify one or more dimensions of graphical representation118(the height and/or width of graphical representation118in pixel values (e.g., 150 pixels by 150 pixels), an offset value representative of an offset of graphical representation in pixel values (e.g., 50 pixels by 50 pixels) from an origin coordinate of a display region (e.g., the top left pixel or coordinate (0,0)) of the application window in which graphical representation118is to be displayed). The encoding may further specify characteristics of the application window of application112in which graphical representation118is included. For example, the encoding may specify dimensions of the display region of the application (e.g., the height and/or width of the display region in pixel values).

Code114may be configured to dynamically generate a new graphical representation118each time the application window resized. The new graphical representation118comprises an encoding that reflects the dimensions of the resized application window. For instance, code114may be configured to detect one or more events issued by an operating system executing on computing device102and/or application(s)114when the application window is resized. In an embodiment in which code114comprises JavaScript, the foregoing may be achieved by utilizing the addEventListener method (e.g., window.addEventListener(“resize,” displayWindowSize). In accordance with an embodiment, graphical representation118is a quick response (QR) code (e.g., a real time and responsive QR code).

External rendering device104is configured to receive video signal110(e.g., via one of the aforementioned communication interfaces). External rendering device104is configured to analyze video signal110to detect graphical representation118. Upon detecting graphical representation118, external rendering device104analyzes the encoding of graphical representation118to determine the network address, the characteristics of graphical representation118, and/or the characteristics of the application window in which graphical representation118is included.

External rendering device104is configured to access and/or retrieve content item108via network116in accordance with the network address specified by the encoding of graphical representation118and render content item108for display by display device106. For instance, external rendering device104may render content item108such that it is placed in the application window in which graphical representation118is displayed. Content item108is rendered based on the characteristics (e.g., dimensions) determined for the application window and/or the characteristics (e.g., dimensions) determined for graphical representation118. For instance, server120may scale content item108in accordance with the video characteristics (e.g., the video resolution and/or aspect ratio) utilized by computing device102. External rendering device204, however, may have different video characteristics. To ensure that content item108is rendered properly, external rendering device104may scale content item108to synchronize the video resolution of the incoming video signal (i.e., video signal110) and the video resolution utilized by external rendering device104utilizing the characteristics of the application window and the characteristics of graphical representation118. This advantageously causes content item108to not appear skewed or distorted when displayed by display device106. External rendering device104generates a video signal122, which comprises the video data provided via video signal110, as well as the video data corresponding to content item108. Video signal122is provided to display device106, which displays content item108rendered by external rendering device104over the application window of the application of application(s)112that requested content item108.

As shown inFIG.1and described below with reference toFIG.2, external rendering device104may comprise a network interface via which external rendering device104communicates via network116. However, it is noted that the embodiments described herein are not so limited. For instance, external rendering device104may utilize the network connection established between computing device102and server120to retrieve content item108. In such an example, external rendering device104may provide a request for content108to computing device102and computing device102may retrieve content item108on behalf of external rendering device104.

It is also noted that the embodiments described herein are not limited to rendering and displaying secure content items associated with remote applications (e.g., via server120). For instance, secure content items may be associated with application(s) (e.g., application(s)112) or process(es) executing on computing device102. In accordance with such embodiments, code114originates from such application(s) or process(es) (rather than server120). Computing device102provides graphical representation118to external rendering device104, as described above, and external rendering device104determines a location (e.g., a network address) at which the secure content item is located based on the encoding of graphical representation118. The secure content item may be stored on a remote server or on computing device102. In the latter case, the location specified by the encoding may specify a network address of computing device102, and external rendering device104may retrieve the secure content item from computing device102via a network interface of external rendering device104and network116.

In accordance with the foregoing techniques, only individuals and/or organizations that utilize external rendering device104, which is configured to detect and analyze graphical representation120and render content item108outside of the traditional graphic pipeline, are able to view content item108. For instance, server120may be configured to authenticate external rendering device104before providing content item108. For example, when establishing a connection with server120and/or providing a request for content item108, external rendering device104may provide an attestation key to server120, which server120utilizes to verify whether external rendering device120is authorized to communicate with server120and/or authorized to receive content item120. If server120determines that external rendering device120is authorized, server120provides content item108responsive to receiving a request therefor from external rendering device104. If server120determines that external rendering device120is not authorized, then server120does not provide content108to external rendering device104. This advantageously enables only approved external rendering devices (e.g., external rendering device104) to retrieve secured content items (e.g., content item108).

FIG.2depicts a block diagram of a system200configured to render and display a secure content item via a rendering device externally coupled to a computing device in accordance with another example embodiment. As shown inFIG.2, system200comprises a computing device202, an external rendering device204, a display device206, and a server220. Computing device202, external rendering device204, display device206, and server220are examples of computing device102, external rendering device104, a display device106, and server120, as described above with reference toFIG.1. Computing device202and external rendering device204are communicatively coupled to server220via network216, which is an example of network116, as described above with reference toFIG.1.

As further shown inFIG.2, computing device202comprises application(s)212, which are examples of application(s)112, as described above with reference toFIG.1. An application of application(s)212may issue a request221for a content item208hosted by server220. Content item208is an example of content item108, as described above with reference toFIG.1. Request221may be an HTTP (hypertext transfer protocol)-based message, such as an HTTP GET request, a Remote Desktop Protocol (RDP)-based request, an RDP over HTTP Secure (HTTPS)-based request, or other network-based or remote desktop-based protocol requests. In response to receiving request221, server220provides code214rather than content item208via a response224, such as an HTTP-based response, an RDP-based response, an RDP over HTTPS-based response, etc. Code214is an example of code114, as described above with reference toFIG.1.

The requesting application is configured to execute code214. Execution of code214results in a graphical representation218of content item208, which is rendered by computing device202(e.g., a processor or GPU of computing device202). Graphical representation218may be rendered in an application window of the requesting application.

Computing device202provides graphical representation218(and other content being rendered by computing device202, e.g., a desktop of an operating system, other application windows, user interface elements, etc.) via a video signal210. For instance, computing device202may comprise a video output interface by which video signal210is transmitted. Examples of a video output interfaces include, but are not limited to an HDMI port, a DVI port, USB port, an I2C port, PCIe port, a DP port, an eDP port, etc. A device comprising a corresponding video input interface (e.g., an HDMI port, a DVI port, USB port, an I2C port, PCIe port, a DP port, an eDP port, etc.) is configured to receive, process and display video signal210.

For example,FIG.3Adepicts an example application window300in accordance with an example embodiment. In the example shown inFIG.3A, the application requesting content item208is a browser application. Accordingly, application window300is a browser application window. However, it is noted that other types of applications may request for content item208.

As shown inFIG.3A, application window300comprises a user interface304and a display region302in which various content is rendered and displayed (e.g., web pages, documents, images, videos, content item208, an application window and/or user interface elements (e.g., in an embodiment in which a remote or virtual desktop is being provided via server120), graphical representation218). User interface304is a graphical user interface (GUI) that comprises a plurality of user interface elements. Examples of such user interface elements include, but are not limited to, an address bar306for specifying a network address to which the browser application is to navigate (e.g., a network address associated with server220) and to display the network address of a current web page navigated to by the browser application, a back button and a forward button, which enable a user to navigate to different network addresses stored in a browser history maintained by the browser application in a backward and forward manner, respectively, a refresh button, which reloads or (“refreshes”) the web page displayed in application window300, etc.

As further shown inFIG.3A, a graphical representation318is rendered and displayed in display302of application window300. Graphical representation318is an example of graphical representation218, as described above with reference toFIG.2. In the example shown inFIG.3A, graphical representation318is a QR code. Such QR codes comprise a relatively small amount of data (e.g., up to 3 Kilobytes). Accordingly, a small amount of data is required to be transmitted to external rendering device204and processed by external rendering device204to retrieve and render content item208. Accordingly, the embodiments described herein minimize the expenditure of computing resources (e.g., input/output operations, processor cycles, memory, power, etc.) for both computing device202that provides graphical representation218and external rendering device204that analyzes graphical representation218and retrieves and renders content item208. It is noted that other types of graphical representations in addition to and/or in lieu of QR codes may be utilized.

Graphical representation318utilizes an encoding to specify a network address at which content item208may be retrieved. The encoding may also specify various characteristics of graphical representation318itself. For instance, the encoding may specify one or more dimensions of graphical representation118(the height and/or width of graphical representation218in pixel values (e.g., 150 pixels by 150 pixels), an offset value representative of an offset of graphical representation in pixel values (e.g., 50 pixels by 50 pixels) from a coordinate of a display region (e.g., an origin coordinate307representative of the top left pixel of display region302(i.e., coordinate (0,0)). The encoding may further specify characteristics of application window300. For example, the encoding may specify a first dimension value representative of a first dimension310(e.g., the height) of display region302of application window300and a second dimension value representative of a second dimension312(e.g., the width) of display region302of application window300.

Graphical representation318is displayed to a user via application300when external rendering device304is not being utilized or before external rendering device304has analyzed the contents of graphical representation218and/or retrieved and rendered the associated content item (e.g., content item208). As described below with reference toFIGS.9-11C, graphical representation318may be displayed when application window300is inactive.

Referring again toFIG.2, external rendering device204may include one or more of processors226, one or more of a memory and/or other physical storage device (“memory”)228, a network interface236, a video input interface230, a video output interface234, and a graphical representation analyzer232. External rendering device204may also include additional components (not shown for brevity and illustrative clarity) including, but not limited to, components and subcomponents of other devices and/or systems herein, as well as those described below with respect toFIGS.12and13.

Processor(s)226and memory228may respectively be any type of processor circuit and memory that is described herein, and/or as would be understood by a person of skill in the relevant art(s) having the benefit of this disclosure. Processor(s)226and memory228may each respectively comprise one or more processors or memories, different types of processors or memories, remote processors or memories, and/or distributed processors or memories. Processor(s)226comprise circuitry that is configured to execute computer program instructions such as but not limited to embodiments of graphical representation analyzer232, which may be implemented as computer program instructions for rendering secured content items, etc., as described herein. In other embodiments, graphical representation analyzer232may be implemented by circuitry of processor(s)226or another circuit communicatively coupled to processor(s)226. Memory228is configured to store such computer program instructions/code, as well as to store other information and data described in this disclosure including, without limitation, characteristics of content item208, characteristics of the application window in which content item208is to be displayed, frame buffers, etc. In still other embodiments, external rendering device204may be implemented as a thin client configured with a simple or low-performance processor and network connectivity. In accordance with such embodiments, external rendering device204establishes a network connection (e.g., via network interface236) with a remote server that is configured to perform the operations of graphical representation analyzer232, as described herein.

Video input interface230may comprise hardware and/or software and is configured to receive video signals from any number and/or types of source devices (e.g., computing device202) configured to output video signals. For example, video input interface230may comprise one or more ports configured to receive a video cable utilized for transmission of video signals. In a particular example, the video cable couples external rendering device204to computing device202. Examples of port(s) include, but are not limited to, HDMI ports, USB ports, PCIe ports, I2C ports, DVI ports, DP ports, eDP ports, etc. Examples of video cables include, but are not limited to, HDMI cables, USB cables, PCIe cables, I2C cables, DVI cables, DP cables, eDP cables, etc.

Video output interface234may comprise hardware and/or software and is configured to output video signals to any number and/or types of sink devices (e.g., display device206) configured to display video signals. For example, video output interface234may comprise one or more ports configured to receive a video cable utilized for transmission of video signals. In a particular example, the video cable couples external rendering device204to display device206. Examples of port(s) include, but are not limited to, HDMI ports, USB ports, PCIe ports, I2C ports, DVI ports, DP ports, eDP ports, etc. Examples of video cables include, but are not limited to, HDMI cables, USB cables, PCIe cables, I2C cables, DVI cables, DP cables, eDP cables, etc.

Network interface236may be any type or number of wired and/or wireless network adapter, modem, etc., configured to enable external rendering device204to communicate with other devices (such as server220) over network216.

Video input interface230is configured to receive video signal210and provide video signal210to graphical representation analyzer232. Graphical representation analyzer232is configured to analyze video signal210to detect graphical representation218included therein. For instance, the encoding of graphical representation218may comprise an identifier that indicates that graphical representation218is intended for external rendering device204. The identifier may specify an alpha and/or numeric identifier that is specifically associated with external rendering device204. Graphical representation analyzer232may compare the identifier to an identifier maintained locally, for example, in memory228. Upon determining that the identifiers match, graphical representation analyzer232determines that graphical representation218included in video signal218is to be further analyzed and rendered by external rendering device204. If the identifiers do not match or no such an identifier is included, then graphical representation analyzer232acts as a pass-through and provides video signal210to video output interface234, which outputs video signal210to display device206for display thereby.

Upon determining that the identifier that is encoded in graphical representation218matches the identifier maintained by external rendering device204, graphical representation analyzer232further analyzes the encoding of graphical representation218to determine the network address at which content item208is located, the characteristics of graphical representation218, and/or the characteristics of the application window (e.g., application window300, as shown inFIG.3A) in which graphical representation218is included.

Graphical representation analyzer232is configured to retrieve content item208via network interface236and network216. For instance, graphical representation analyzer232may issue a request238for content item208. Request238may be an HTTP-based message, such as an HTTP GET request, an RDP-based request, an RDP over HTTPS-based request, etc. Network interface236is configured to provide request238to a server specified by the network address included in the encoding of graphical representation218. In the example shown inFIG.2, the server is server220; however, content item208may be maintained by a server other than server220. Server220returns a response240, such as an HTTP-based response, an RDP-based response, an RDP over HTTPS-based response, etc., comprising content item208via network216. Network interface236is configured to receive response240. Graphical representation analyzer232may be configured to analyze response240and obtain content item208from response240.

Graphical representation analyzer232is also configured to determine a scaling factor, which is utilized to synchronize the video resolution utilized by computing device202and the video resolution utilized by external rendering device204. This advantageously causes content item208to not appear skewed or distorted when displayed by display device106. To determine the scaling factor, graphical representation analyzer232determines one or more dimensions of graphical representation218(the height and/or width) of graphical representation218in pixel values as would be displayed in the display region of the application window by display device206. Note that these dimensions are not the same dimensions that are encoded in graphical representation218(although the values of such dimensions may be the same in certain scenarios, for example, when the video resolutions of external rendering device204and computing device202match). Instead, these dimensions are determined based on an analysis of the data transmitted via video signal210. For instance, the data transmitted via video signal210may be stored via a frame buffer (also referred herein as the main frame buffer) rendered by computing device202and maintained by external rendering device204, e.g., stored in memory228. The frame buffer comprises data that describes all the visual elements that will appear on display device206, such as menus, application windows, images, videos, etc. For example, the frame buffer comprises a bitmap comprising pixel values for all the pixels that drive display device206. Graphical representation analyzer232may be configured to reconstruct and/or analyze the frame buffer to determine the dimensions of graphical representation218, as would be displayed via display device206. For example, with reference toFIG.3A, graphical representation analyzer232may determine a first dimension value representative of a first dimension314(e.g., the height) of graphical representation318and a second dimension value representative of a second dimension316(e.g., the width) of graphical representation318.

Referring again toFIG.2, graphical representation analyzer232determines the scaling factor based on at least one of the dimensions of graphical representation318that is encoded therein and at least one of the dimensions of graphical representation318determined via analysis of the frame buffer (i.e., the dimensions(s) of graphical representation318as would be displayed via display device206). In accordance with an embodiment, the scaling factor may be determined in by dividing the width value (e.g., dimension316, as shown inFIG.3A) of graphical representation218as would be displayed by display device206by the width value of graphical representation218that is encoded thereby. In accordance with another embodiment, the scaling factor may be determined in by dividing the height value (e.g., dimension314, as shown inFIG.3A) of graphical representation218as would be displayed by display device206by the height value of graphical representation218that is encoded thereby. It is noted that in embodiments in which graphical representation218comprises a square shape, the respective height and width values are the same value. In such embodiments, the encoding of graphical representation218may specify a single dimension value, as both the height and width values are the same.

Graphical representation analyzer232then utilizes the scaling factor to determine the dimensions of content item208to be displayed by display device206. For instance, graphical representation analyzer232may scale the dimension(s) of the display region (e.g., display region302, as shown inFIG.3A) of the application window (e.g., application window300, as shown inFIG.3A) in which content item208is to be displayed based on scaling factor. In accordance with one embodiment, graphical representation analyzer232may multiply the scaling factor by a first dimension determined for the display region (e.g., width312, as shown inFIG.3A) to determine a corresponding dimension (i.e., width) for content item208, and graphical representation analyzer232may multiply the scaling factor by a second dimension determined for the display region (e.g., height310, as shown inFIG.3A) to determine a corresponding dimension (i.e., height) for content item208.

To determine the location at which content item208should be displayed, graphical representation analyzer232analyzes the frame buffer to determine the origin coordinate of the display region in which content item208is to be displayed (e.g., origin coordinate307, as shown inFIG.3A). To determine the origin coordinate, graphical representation analyzer232may analyze the frame buffer to determine a particular coordinate at which graphical representation218is located with respect to the entire image (or view) displayed by display device206. For instance, graphical representation analyzer232may determine the coordinate at which the top left corner of graphical representation218is located (it is noted that any corner of graphical representation218may be utilized). Graphical representation analyzer232may then subtract the offset value (as determined from the encoding of graphical representation218) from the coordinate at which the top left corner of graphical representation218is located. The resulting value corresponds to the origin coordinate of the display region of the application window. Graphical representation analyzer232then scales the x and y coordinates of the determined origin coordinate based on the scaling factor. For instance, graphical representation analyzer232may multiply the x coordinate and they coordinate of the determined origin coordinate by the scaling factor to determine the corresponding x and y coordinates at which the top left of content item208is to be displayed.

Utilizing the scaled x and y coordinates and the scaled dimensions, graphical representation analyzer232renders content item208into another frame buffer, which corresponds to the location and size of the display region of the application window in which content item208is to be displayed. This frame buffer comprises a bitmap comprising pixel values for all the pixels corresponding to the scaled content item208. Graphical representation analyzer232overlays the frame buffer comprising the scaled content item208on top of a portion of the main frame buffer corresponding to the display region of the application window in which the scaled content item208is to be displayed by display device206. Graphical representation analyzer232renders the frame buffer comprising the scaled content item208and outputs a video signal222comprising the content rendered by computing device202and the scaled content item208. Video signal222is an example of video signal122, as described above with reference toFIG.1. Video signal222is provided to display device206for display thereby via video output interface234.

In accordance with an embodiment, graphical representation analyzer232may replace the pixel values of the pixels that comprise the display region of the application window with the pixel values of the pixels that comprise the scaled content item208. Graphical representation analyzer232may convert the resulting bitmap to video signal222.

The foregoing techniques cause the scaled content item208to be displayed over the display region of the application window, thereby causing the scaled content item208to be appear as if it is actually rendered within the application window by computing device202, even though it was rendered by external rendering device204. The foregoing techniques advantageously enables a seamless integration of the content rendered by external rendering device204and the content rendered by computing device202.

For example,FIG.3Bdepicts application window300after a secured content item has been rendered by external rendering device204in accordance with an example embodiment. As shown inFIG.3B, secured content item308is displayed in display region302of application window300. Content item308is an example of content item208, as described above with reference toFIG.3. In the example shown inFIG.3B, a browser application has navigated to a banking website, and content item308is a website comprising a user's personal banking information made available via the banking website. As further shown inFIG.3B, content item308appears to be rendered by computing device202(as described above with reference toFIG.2) and displayed within display region302of application window300when in fact it was rendered by external rendering device204(as described above with reference toFIG.2) and displayed over display region302.

Accordingly, secured content items may be rendered and displayed in many ways. For instance,FIG.4shows a flowchart400for rendering, by a device configured to be externally coupled to a computing device, a secured content item in accordance with an example embodiment. In an embodiment, flowchart400may be implemented by external rendering device204, as described above with reference toFIG.2, although the method is not limited to that implementation. Accordingly,FIG.4will be described with continued reference toFIG.2. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart400and external rendering device204ofFIG.2.

Flowchart400begins at step402. In step402, a video signal comprising a graphical representation of content to be rendered in a display region of an application window displayed via a display device coupled to the device configured to be externally coupled to the computing device is received from the computing device. For example, with reference toFIG.2, video input interface230of external rendering device receives video signal210from computing device202. Video signal210comprises graphical representation218, which is to be rendered in a display region (e.g., display region302, as shown inFIG.3A) of an application window (e.g., application window300, as shown inFIG.3A) displayed via display device206.

In accordance with one or more embodiments, the device configured to be externally coupled to the computing device is a docking station. For example, with reference toFIG.2, external rendering device204may be a docking station.

In accordance with one or more embodiments, the device configured to be externally coupled to the computing device is incorporated into the display device. For example, with reference toFIG.2, external rendering device204may be incorporated into display device206.

In accordance with one or more embodiments, the graphical representation is a QR code. For example, with reference toFIG.2, graphical representation218may be a QR code. As described above, QR codes comprise a relatively small amount of data (e.g., up to 3 Kilobytes). Accordingly, a small amount of data is required to be transmitted to external rendering device204and processed by external rendering device204to retrieve and render content item208. Accordingly, the embodiments described herein also minimize the expenditure of computing resources (e.g., input/output operations, processor cycles, memory, power, etc.) for both computing device202that provides graphical representation218and external rendering device204that analyzes graphical representation218and retrieves and renders content item208.

In step404, the graphical representation is analyzed to determine characteristics of the display region of the application window, characteristics of the graphical representation, and a network address at which the content is located. For example, with reference toFIG.2, video input interface230provides video signal210to graphical representation analyzer232, which is configured to analyze graphical representation218included in video signal210to determine characteristics of the display region (e.g., display region302, as shown inFIG.3A) of the application window (e.g., application window300, as shown inFIG.3A), characteristics of graphical representation218, and a network address at which the content (e.g., content item208) is located.

In accordance with one or more embodiments, the characteristics of the display region of the application window comprise at least one of a first dimension value representative of a first dimension of the display region of the application window or a second dimension value representative of a second dimension of the display region of the application window, and the characteristics of the graphical representation comprise at least one of an offset value representative of an offset of the graphical representation from a first origin coordinate of the display region of the application window, a third dimension value representative of a third dimension of the graphical representation, or a fourth dimension value representative of a fourth of the graphical representation. For example, with reference toFIG.3A, the characteristics of display region302of application window300comprise at least one of a first dimension value representative of a first dimension310(e.g., the height) of display region302of application window300or a second dimension value representative of a second dimension312(e.g., the width) of display region302of application window300, and the characteristics of graphical representation318comprise at least one of an offset value representative of an offset of graphical representation318from a first origin coordinate (e.g., coordinate308) of display region302of application window300, a third dimension value representative of a third dimension (e.g., the height) of graphical representation, or a fourth dimension value representative of a fourth dimension (e.g., the width) of graphical representation318. As described below with reference toFIG.7, the first dimension value and the second dimension value may be scaled in accordance with a scaling factor to synchronize the video resolution of the incoming video signal (e.g., video signal210, as shown inFIG.2) and the video resolution utilized by external rendering device204. This advantageously causes content item208to not appear skewed or distorted. As also described below, the third dimension value and the fourth dimension value may be utilized to determine a coordinate at which the scaled version of content item208is to be displayed with respect to the display region. In particular, content item208is positioned such that it appears over the display region. This advantageously provides a seamless experience in which it appears that content item208is rendered within the display region of the application window (and rendered by computing device202), even though content item208is rendered over the display region (and rendered by external rendering device204).

In step406, the content is retrieved based on the network address. For example, with reference toFIG.2, graphical representation analyzer232generates a request238for content item208, which is provided to a server (e.g., server220) at which content item208is located. Request238is provided to network interface236, which provides request238to server220via network216.

In step408, the content is rendered over the display region of the application window in accordance with the characteristics determined for the display region of the application window and the characteristics determined for the graphical representation. The resulting video signal (e.g., video signal222) is provided to display device206for display thereby via video output interface234. For example, with reference toFIG.2, graphical representation analyzer232renders content item208over the display region of the application window. For instance, as shown inFIG.3B, content item308is displayed over display region302of application window300.

By having external rendering device204retrieve and render content item208(as opposed to computing device202), the embodiments described herein advantageously prevent secured content items (e.g., content item208) from being rendered and displayed when rendering device204is not coupled to computing device202, as the secure content item is not provided to computing device202for rendering thereby. Instead, the rendering of the secure content item is performed outside of the traditional graphic pipeline; in particular, the rendering is performed by external rendering device204coupled between computing device202and display device206. Accordingly, the techniques described herein prevent access to secured content items, such as personal and/or confidential information, unless external rendering device204is utilized.

In accordance with one or more embodiments, the content is scaled before being rendered over the display region of the application window.FIG.5shows a flowchart500for scaling content in accordance with an example embodiment. In an embodiment, flowchart500may be implemented by an external rendering device600shown inFIG.6. However, it is noted that the method is not limited to that implementation.FIG.6depicts a block diagram of external rendering device600in accordance with an example embodiment. External rendering device600is an example of external rendering device204, as described above with reference toFIG.2. As shown inFIG.6, external rendering device600may comprise a graphical representation analyzer632, a video input interface630and a content item608. Graphical representation analyzer632, video input interface630, and content item608are respective examples of comprise graphical representation analyzer232, video input interface230, and content208, as described above with reference toFIG.2. It is noted that external rendering device600may comprise other components (as described above with reference toFIG.2) that are not shown inFIG.6for the sake of brevity. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart500and external rendering device600ofFIG.6.

Flowchart500begins at step502. In step502, at least one of a fifth dimension value representative of a fifth dimension of the graphical representation as displayed in the display region of the application window or a sixth dimension value representative of a sixth dimension of the graphical representation as displayed in the display region of the application window is determined. For example, with reference toFIG.6, video signal analyzer602may analyze a video signal610received via video input interface630to determine a fifth dimension value representative of a fifth dimension of the graphical representation as displayed in the display region of the application window or a sixth dimension value representative of a sixth dimension of the graphical representation as displayed in the display region of the application window is determined. For example, with reference toFIG.3A, video signal analyzer602may determine a fifth dimension value representative of a fifth dimension314(e.g., the height) of graphical representation318as displayed in display region302of application window300or a sixth dimension value representative of a sixth dimension316(e.g., the width) of graphical representation318as displayed in display region302of application window300. Referring again toFIG.6, the determined dimensions (shown as dimension(s)612) are provided to scaling factor determiner604.

In step504, a scaling factor is determined based on at least one of the first dimension value or the second dimension value and at least one of the fifth dimension value or the sixth dimension value. For example, with reference toFIG.6, scaling factor determined determines a scaling factor614based at least one the first dimension value or the second dimension value (shown as dimension(s)616, which may be determined via step404, as described above with reference toFIG.4) and dimension(s)612. In accordance with an embodiment, scaling factor determiner604may determine scaling factor614by dividing the fifth dimension value of dimension(s)612(e.g., the height) of the graphical representation as displayed by the display device by first dimension value of dimension(s)614(e.g., the height) of the graphical representation that is encoded thereby. In accordance with another embodiment, scaling factor determiner604may determine scaling factor614by dividing the sixth dimension value of dimension(s)612(e.g., the width) of the graphical representation as displayed by the display device by the second dimension value of dimension(s)616(e.g., the width) of the graphical representation that is encoded thereby. It is noted that in embodiments in which the graphical representation comprises a square shape, the respective height and width values are the same value. In such embodiments, the encoding of the graphical representation may specify a single dimension value, as both the height and width values are the same. Scaling factor614is provided to scaler606.

In step506, the content is scaled in accordance with the scaling factor. For example, with reference toFIG.6, scaler606scales content item608in accordance with scaling factor612. The scaled content (shown as scaled content item618) is provided to renderer620. As described herein, the scaling factor determined using the third dimension value, the fourth dimension value, the fifth dimension value, and the sixth dimension value is utilized to scale the dimensions (e.g., the first dimension value and the second dimension value) is utilized to scale content item608. This advantageously causes content item208to not appear skewed or distorted when displayed over the display region of an application window. Additional details regarding scaling the content are described below with reference toFIGS.7and8.

In step508, the scaled content is rendered over the display region of the application window. For example, with reference toFIGS.3B and6, renderer620renders scaled content item618(shown as content item308inFIG.3B) over display region302of application window300. In accordance with an embodiment, renderer620renders scaled content item618into a frame buffer, which corresponds to the location and size of display region302of application window300. The frame buffer comprises a bitmap comprising pixel values for all the pixels corresponding to scaled content item618. Renderer620overlays the frame buffer comprising scaled content item618on top of a portion of the main frame buffer corresponding to display region302of application window300. Renderer620renders the frame buffer comprising scaled content item318and outputs a video signal622comprising the content provided via video signal610and scaled content item618. Video signal622is an example of video signal222, as described above with reference toFIG.2.

In accordance with an embodiment, graphical representation analyzer232may replace the pixel values of the pixels that comprise the display region of the application window with the pixel values of the pixels that comprise the scaled content item208. Graphical representation analyzer232may convert the resulting bitmap to video signal222.

FIG.7shows a flowchart700for determining the size of and location at which content is to be rendered in accordance with an example embodiment. In an embodiment, flowchart700may be implemented by an external rendering device800shown inFIG.8. However, it is noted that the method is not limited to that implementation.FIG.8depicts a block diagram of external rendering device800in accordance with another example embodiment. External rendering device800is an example of external rendering device600, as described above with reference toFIG.6. As shown inFIG.8, external rendering device800may comprise a graphical representation analyzer832, which comprises a scaler806, a renderer820, and a content item808, which are respective examples of external rendering device600, scaler606, renderer620, and content item608, as described above with reference toFIG.6. It is noted that external rendering device800may comprise other components (as described above with reference toFIG.2) that are not shown inFIG.8for the sake of brevity. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart700and external rendering device800ofFIG.8.

Flowchart700begins at step702. In step702, the first dimension value and the second dimension value of the display region of the application window are scaled in accordance with the scaling factor to determine dimensions of the content to be rendered over the display region of the application window. For example, with reference toFIG.8, scaling logic802is configured to scale the first dimension value and the second dimension value (shown as dimension(s)816, which may be determined via step404, as described above with reference toFIG.4) of the display region of the application window in accordance with a scaling factor814. Dimension(s)816and scaling factor814are respective examples of dimensions(s)616and scaling factor614, as described above with reference toFIG.6. Scaling logic802may generate a scaled version of content item808(shown as scaled content item818) and provide scaled content item818to renderer820. Scaled content item818is an example of scaled content item618, as described above with reference toFIG.6. In accordance with an embodiment, scaling logic802scales dimension(s)816by multiplying each of dimension(s)816by scaling factor814. The scaled dimension(s) (shown as scaled dimension(s)826) are provided to renderer820. The first dimension value and the second dimension value are scaled in accordance with scaling factor614to synchronize the video resolution of the incoming video signal (i.e., video signal810) and the video resolution utilized by external rendering device800utilizing. This advantageously content item808to not appear skewed or distorted.

In step704, a second origin coordinate is determined based on a coordinate at which the graphical representation is located with respect to the display device and the offset value. For example, with reference toFIGS.3A and8, location determiner804is configured to determine a second origin coordinate based on a coordinate of graphical representation318(e.g., the top left coordinate of graphical representation318) with respect to the display device (e.g., with respect to the entire image displayed by the display device) and an offset value810. Offset value810may be specified via the encoding of graphical representation318and may be determined via step404, as described above with reference toFIG.4. In accordance with an embodiment, location determiner804may determine the x coordinate of the second origin coordinate by subtracting offset value810from the x coordinate at which the top left corner of graphical representation318is located with respect to the entire image (or view) displayed by the display device and determine they coordinate of the second origin coordinate by subtracting offset value810from the y coordinate at which the top left corner of graphical representation318is located with respect to the entire image (or view displayed by the display device. The resulting value corresponds to origin coordinate307of display region302of application window300. The second origin coordinate (shown as coordinate821) is provided to scaling logic802.

In step706, the second origin coordinate is scaled to determine a location at which the scaled content is to be rendered over the display region of the application window. For example, with reference toFIG.8, scaling logic802scales the x and y coordinates of the coordinate821based on scaling factor816. For instance, scaling logic802may multiply the x coordinate and the y coordinate of coordinate821by scaling factor816to determine the corresponding x and y coordinates at which the top left of content item808is to be displayed. The scaled coordinate (shown as scaled coordinate824) is provided to renderer820. Accordingly, offset value810is advantageously utilized to determine a coordinate (i.e., scaled coordinate824) at which scaled content item818is to be displayed with respect to the display region. In particular, scaled content item818is positioned such that it appears over the display region. This advantageously provides a seamless experience in which it appears that scaled content item818is rendered within the display region of the application window (and rendered by the computing device that provides video signal810), even though scaled content item818is rendered over the display region (and rendered by external rendering device800).

Utilizing the scaled coordinate824and/or scaled dimension(s)826, renderer820renders scaled content item818into a frame buffer, which corresponds to the location and size of the display region of the application window in which scaled content item818is to be displayed. This frame buffer comprises a bitmap comprising pixel values for all the pixels corresponding to the scaled content item818. Renderer820overlays the frame buffer comprising scaled content item818on top of a portion of the main frame buffer corresponding to the display region of the application window in which scaled content item818is to be displayed. Renderer820renders the frame buffer comprising scaled content item818and outputs a video signal822, which is an example of video signal622.

In accordance with an embodiment, renderer820may replace the pixel values of the pixels that comprise the display region of the application window with the pixel values of the pixels that comprise scaled content item818. Renderer820may convert the resulting bitmap to video signal822.

In accordance with an embodiment, secured content items are only displayed in an application window responsive to detecting a particular triggering event. In one example, the triggering event is detecting that the application window in which the secured content item is to be displayed is active; however, it is noted that other types of triggering events may be utilized. In accordance with such an embodiment, the graphical representation encoding may specify additional characteristics associated with the application window. For instance, the encoding may specify an application window identifier that uniquely identifies the application window in which the secured content item is to be displayed, a state value that indicates whether a cursor rendered by the host computing device is inside the display region of the application window, an x-axis position value that indicates the x coordinate of the cursor (e.g., in pixels) inside the display region of the application window relative to the origin coordinate, and a y-axis position value that indicates the y coordinate of the cursor (e.g., in pixels) inside the display region of the application window relative to the origin coordinate.

As described above, the graphical representation is dynamically generated by code (e.g., code114) when the application window is resized. In accordance with the foregoing embodiment, the graphical representation is also dynamically generated by the code each time the cursor is moved to a new position within the display region of the application window, each time the cursor is placed within the display region and/or each time the cursor leaves the display region.

Utilizing the state value, the external rendering device (e.g., external rendering device104, as shown inFIG.1) determines whether or not the application window is in an active state. For example, if the state value indicates that a curser is inside the display region of the application window, then the external rendering device determines that the application window is in an active state (i.e., the user is utilizing and/or viewing the content of application window). Otherwise, the external rendering device determines that the application window is not in an active state. Responsive to determining that the application window is active, the external rendering device renders the secured content into the window identified by the application window identifier. When a user subsequently moves the curser such that it is no longer in the display region of the application window, the encoding of the graphical representation is updated to indicate as such, and the external rendering device no longer renders the secured content item (and instead, reverts to causing the graphical representation of the content item to be displayed). It is further noted that the external rendering device no longer renders the secured content item responsive to determining that the graphical representation thereof is no longer displayed, e.g., when the application window in which the graphical representation is minimized or obfuscated (e.g., covered) by another application window.

Because the secured content item is rendered over the display region by the graphical representation analyzer232, the cursor generated by the operating system executing on the host computing device is no longer displayed. This causes a disruption in the user's ability to interact with the secured content item. To overcome this issue, the external rendering device may render a cursor located at the cursor's x and y coordinates, as specified by the encoding in the graphical representation of the content item.

Accordingly, secured content items are only displayed in an application window responsive to detecting a particular triggering event in many ways. For instance,FIG.9shows a flowchart900for displaying a secured content item responsive to detecting a triggering event in accordance with an example embodiment. In an embodiment, flowchart900may be implemented by an external rendering device1000shown inFIG.10, However, it is noted that the method is not limited to that implementation.FIG.10depicts a block diagram of external rendering device1000in accordance with a further example embodiment. External rendering device1000is an example of external rendering device800, as described above with reference toFIG.8. As shown inFIG.10, external rendering device1000may comprise a graphical representation analyzer1032, a video input interface1030, a state determiner1004, and a renderer1020. External rendering device1000, graphical representation analyzer1032, video input interface1030, and renderer1020are respective examples of comprise external rendering device600and800, graphical representation analyzer632and832, video input interface630, and renderer620and820, as described above with reference toFIGS.6and8. It is noted that external rendering device1000may comprise other components (as described above with reference toFIG.2) that are not shown inFIG.6for the sake of brevity. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the discussion regarding flowchart900and external rendering device1000ofFIG.10.

Flowchart900begins at step902. In step902, a triggering event is detected. For example, with reference toFIG.10, state determiner1004detects the triggering event.

In accordance with one or more embodiments, the triggering event comprises detecting a cursor rendered by the computing device is placed within the display region of the application window. For example, with reference toFIG.10, video signal analyzer1002may analyze an encoding of a graphical representation of a secured content item included in video signal1010received via video input interface1030to determine an application window identifier that uniquely identifies the application window in which the secured content item, the state value, the x-axis position value, and the y-axis position value. Such data (shown as data1012) is provided to state determiner1004. State determiner1004is configured to determine whether the application window identified by the application window identifier is in an active state by analyzing state value. If the state value indicates that a cursor is included in the display region of the application window, then state determiner1004determines that the application window is active. In response to determining that the cursor is included in the display region, state determiner1004may output an indication1014that indicates as such. If the state value indicates that a cursor is not included in the display region of the application window, then state determiner1004determines that the application window is inactive. In response to determining that the cursor is included in the display region, state determiner1004may output an indication that indicates as such.

In step904, the content is overlayed on top of the graphical representation responsive to detecting the triggering event. For example, with reference toFIG.10, renderer1020, responsive to receiving indication1014, renderer1020generates a video signal1022that comprises scaled content item1018, which is overlayed on top of the graphical representation. Scaled content item1018is an example of scaled content item818, as described above with reference toFIG.8.

In accordance with an embodiment, renderer1020is further configured to render a cursor in place of the cursor rendered by the host computing device. For instance, renderer1020may render the cursor at the x and y coordinates specified by the encoding in the graphical representation of the content item. In accordance with such an embodiment, data1012(or portions thereof, such as the x and y coordinates determined from the encoding) may be provided to renderer1020. Renderer1020utilizes these coordinates to determine a location at which the cursor should be rendered. The coordinates may be scaled (e.g., by scaler606, as shown inFIG.6) utilizing a scaling factor (e.g., scaling factor614, as shown inFIG.6), and the scaled coordinates may be utilized as the location at which the cursor is to be rendered.

The foregoing techniques is described with reference toFIGS.11A-11C. For example,FIG.11Adepicts an example application window1100in accordance with an example embodiment. Application window1100is an example of application window300, as described above with reference toFIG.3A.

As shown inFIG.11A, application window1100comprises a display region1102, which is an example of display region302, as shown inFIG.3A. As further shown inFIG.11A, a graphical representation1118is rendered and displayed in display region1102of application window1100. Graphical representation1118is an example of graphical representation318, as described above with reference toFIG.3A. Graphical representation1118utilizes an encoding to specify an application window identifier that uniquely identifies application window1100in which secured the content item is to be displayed, a state value that indicates whether a cursor1104rendered by the host computing device is inside display region1102of application window1100, an x-axis position value that indicates the x coordinate of cursor1104(e.g., in pixels) inside display region of the application window relative to its origin coordinate1107(which is an example of origin coordinate307), and a y-axis position value that indicates the y coordinate of cursor1104(e.g., in pixels) inside display region1102of application window1100relative to its origin coordinate1107.

As shown inFIG.11A, cursor1104is located outside of display region1102. Accordingly, the state value encoded via graphical representation1118indicates that the application window1100is inactive. Accordingly, external rendering device1000does not render the secured content item, and instead provides the video signal received from the host computing device to the display device coupled to external rendering device1000, which causes graphical representation1118to be displayed in display region.

As shown inFIG.11B, a user has moved cursor1104over display region1102. As such, graphical representation1118is re-generated such that the state value encoded via the newly-generated graphical representation indicates that the application window1100is active. The encoding also specifies the x and y coordinates of cursor1104with respect to origin coordinate1107. In response to detecting that application window1100is active, renderer1020causes secured content item1108to be displayed over display region1102, thereby effectively covering the newly-generated graphical representation and cursor1104. Renderer1020utilizes the x and y coordinates specified by the encoding of the newly-generated graphical representation1118to render a cursor (shown as cursor1104′) located at the x and y coordinate.

As shown inFIG.11C, a user has moved cursor1104outside of display region1102. Accordingly, graphical representation1118is re-generated once again such that the state value encoded via graphical representation indicates that application window1100is inactive. Therefore, renderer1020no longer renders secured content item, and the newly-generated graphical representation (shown as graphical representation1118′) is displayed in display region1102.

III. Example Mobile Device and Computing Device Embodiments

Embodiments described herein may be implemented in hardware, or hardware combined with software and/or firmware. For example, embodiments described herein may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium. Alternatively, embodiments described herein may be implemented as hardware logic/electrical circuitry.

As noted herein, the embodiments described, including computing device102, server120, external rendering device104, and/or display device106ofFIG.1, computing device202, server220, external rendering device204, and/or display device206ofFIG.2, external rendering device600ofFIG.6, external rendering device800ofFIG.8and external rendering device1000ofFIG.10, along with any components and/or subcomponents thereof, as well as the flowcharts/flow diagrams described herein, including portions thereof, and/or further examples described herein, may be implemented in hardware, or hardware with any combination of software and/or firmware, including being implemented as computer program code configured to be executed in one or more processors and stored in a computer readable storage medium, or being implemented as hardware logic/electrical circuitry, such as being implemented together in a system-on-chip (SoC), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). A SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or further circuits and/or embedded firmware to perform its functions.

Embodiments described herein may be implemented in one or more computing devices similar to a mobile system and/or a computing device in stationary or mobile computer embodiments, including one or more features of mobile systems and/or computing devices described herein, as well as alternative features. The descriptions of mobile systems and computing devices provided herein are provided for purposes of illustration, and are not intended to be limiting. Embodiments may be implemented in further types of computer systems, as would be known to persons skilled in the relevant art(s).

FIG.12is a block diagram of an exemplary mobile system1200that includes a mobile device1202that may implement embodiments described herein. For example, mobile device1202may be used to implement any system, client, or device, or components/subcomponents thereof, in the preceding sections. As shown inFIG.12, mobile device1202includes a variety of optional hardware and software components. Any component in mobile device1202can communicate with any other component, although not all connections are shown for ease of illustration. Mobile device1202can be any of a variety of computing devices (e.g., cell phone, smart phone, handheld computer, Personal Digital Assistant (PDA), etc.) and can allow wireless two-way communications with one or more mobile communications networks1204, such as a cellular or satellite network, or with a local area or wide area network.

Mobile device1202can include a controller or processor1210(e.g., signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing such tasks as signal coding, data processing, input/output processing, power control, and/or other functions. An operating system1212can control the allocation and usage of the components of mobile device1202and provide support for one or more application programs1214(also referred to as “applications” or “apps”). Application programs1214may include common mobile computing applications (e.g., e-mail applications, calendars, contact managers, web browsers, messaging applications) and any other computing applications (e.g., word processing applications, mapping applications, media player applications).

Mobile device1202can include memory1220. Memory1220can include non-removable memory1222and/or removable memory1224. Non-removable memory1222can include RAM, ROM, flash memory, a hard disk, or other well-known memory devices or technologies. Removable memory1224can include flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM communication systems, or other well-known memory devices or technologies, such as “smart cards.” Memory1220can be used for storing data and/or code for running operating system1212and application programs1214. Example data can include web pages, text, images, sound files, video data, or other data to be sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks. Memory1220can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment.

A number of programs may be stored in memory1220. These programs include operating system1212, one or more application programs1214, and other program modules and program data. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing one or more of code114ofFIG.1, code214and graphical representation analyzer232ofFIG.2, graphical representation analyzer632ofFIG.6ofFIG.6, and graphical representation analyzer832ofFIG.8, along with any components and/or subcomponents thereof, as well as the flowcharts/flow diagrams described herein, including portions thereof, and/or further examples described herein.

Mobile device1202can support one or more input devices1230, such as a touch screen1232, a microphone1234, a camera1236, a physical keyboard1238and/or a trackball1240and one or more output devices1250, such as a speaker1252and a display1254. Other possible output devices (not shown) can include piezoelectric or other haptic output devices. Some devices can serve more than one input/output function. For example, touch screen1232and display1254can be combined in a single input/output device. Input devices1230can include a Natural User Interface (NUI).

One or more wireless modems1260can be coupled to antenna(s) (not shown) and can support two-way communications between processor1210and external devices, as is well understood in the art. Modem1260is shown generically and can include a cellular modem1266for communicating with the mobile communication network1204and/or other radio-based modems (e.g., Bluetooth1264and/or Wi-Fi1262). At least one wireless modem1260is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the mobile device and a public switched telephone network (PSTN).

Mobile device1202can further include at least one input/output port1280, a power supply1282, a satellite navigation system receiver1284, such as a Global Positioning System (GPS) receiver, an accelerometer1286, and/or a physical connector1290, which can be a USB port, IEEE 1394 (FireWire) port, and/or RS-232 port. The illustrated components of mobile device1202are not required or all-inclusive, as any components can be deleted and other components can be added as would be recognized by one skilled in the art.

In an embodiment, mobile device1202is configured to implement any of the above-described features of flowcharts herein. Computer program logic for performing any of the operations, steps, and/or functions described herein may be stored in memory1220and executed by processor1210.

FIG.13depicts an exemplary implementation of a computing device1300in which embodiments may be implemented. For example, embodiments described herein may be implemented in one or more computing devices similar to computing device1300in stationary or mobile computer embodiments, including one or more features of computing device1300and/or alternative features. The description of computing device1300provided herein is provided for purposes of illustration, and is not intended to be limiting. Embodiments may be implemented in further types of computer systems and/or game consoles, etc., as would be known to persons skilled in the relevant art(s).

As shown inFIG.13, computing device1300includes one or more processors, referred to as processor circuit1302, a system memory1304, and a bus1306that couples various system components including system memory1304to processor circuit1302. Processor circuit1302is an electrical and/or optical circuit implemented in one or more physical hardware electrical circuit device elements and/or integrated circuit devices (semiconductor material chips or dies) as a central processing unit (CPU), a microcontroller, a microprocessor, and/or other physical hardware processor circuit. Processor circuit1302may execute program code stored in a computer readable medium, such as program code of operating system1330, application programs1332, other programs1334, etc. Bus1306represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. System memory1304includes read only memory (ROM)1308and random access memory (RAM)1310. A basic input/output system1312(BIOS) is stored in ROM1308.

Computing device1300also has one or more of the following drives: a hard disk drive1314for reading from and writing to a hard disk, a magnetic disk drive1316for reading from or writing to a removable magnetic disk1318, and an optical disk drive1320for reading from or writing to a removable optical disk1322such as a CD ROM, DVD ROM, or other optical media. Hard disk drive1314, magnetic disk drive1316, and optical disk drive1320are connected to bus1306by a hard disk drive interface1324, a magnetic disk drive interface1326, and an optical drive interface1328, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computer. Although a hard disk, a removable magnetic disk and a removable optical disk are described, other types of hardware-based computer-readable storage media can be used to store data, such as flash memory cards, digital video disks, RAMs, ROMs, and other hardware storage media.

A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include operating system1330, one or more application programs1332, other programs1334, and program data1336. Application programs1332or other programs1334may include, for example, computer program logic (e.g., computer program code or instructions) for implementing embodiments described herein, such as one or more of code114ofFIG.1, code214and graphical representation analyzer232ofFIG.2, graphical representation analyzer632ofFIG.6ofFIG.6, and graphical representation analyzer832ofFIG.8, along with any components and/or subcomponents thereof, as well as the flowcharts/flow diagrams described herein, including portions thereof, and/or further examples described herein.

A user may enter commands and information into the computing device1300through input devices such as keyboard1338and pointing device1340. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, a touch screen and/or touch pad, a voice recognition system to receive voice input, a gesture recognition system to receive gesture input, or the like. These and other input devices are often connected to processor circuit1302through a serial port interface1342that is coupled to bus1306, but may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB).

A display screen1344is also connected to bus1306via an interface, such as a video adapter1346. Display screen1344may be external to, or incorporated in computing device1300. Display screen1344may display information, as well as being a user interface for receiving user commands and/or other information (e.g., by touch, finger gestures, virtual keyboard, etc.). In addition to display screen1344, computing device1300may include other peripheral output devices (not shown) such as speakers and printers.

Computing device1300is connected to a network1348(e.g., the Internet) through an adaptor or network interface1350, a modem1352, or other means for establishing communications over the network. Modem1352, which may be internal or external, may be connected to bus1306via serial port interface1342, as shown inFIG.13, or may be connected to bus1306using another interface type, including a parallel interface.

As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium,” etc., are used to refer to physical hardware media. Examples of such physical hardware media include the hard disk associated with hard disk drive1314, removable magnetic disk1318, removable optical disk1322, other physical hardware media such as RAMs, ROMs, flash memory cards, digital video disks, zip disks, MEMs, nanotechnology-based storage devices, and further types of physical/tangible hardware storage media (including memory1320ofFIG.13). Such computer-readable media and/or storage media are distinguished from and non-overlapping with communication media and propagating signals (do not include communication media and propagating signals). Communication media embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wireless media such as acoustic, RF, infrared and other wireless media, as well as wired media. Embodiments are also directed to such communication media that are separate and non-overlapping with embodiments directed to computer-readable storage media.

As noted above, computer programs and modules (including application programs1332and other programs1334) may be stored on the hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage medium. Such computer programs may also be received via network interface1350, serial port interface1342, or any other interface type. Such computer programs, when executed or loaded by an application, enable computing device1300to implement features of embodiments discussed herein. Accordingly, such computer programs represent controllers of the computing device1300.

Embodiments are also directed to computer program products comprising computer code or instructions stored on any computer-readable medium or computer-readable storage medium. Such computer program products include hard disk drives, optical disk drives, memory device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

IV. Further Example Embodiments

An apparatus configured to be externally coupled to a computing device. The apparatus comprises: at least one processor circuit; and at least one memory that stores program code configured to be executed by the at least one processor circuit, the program code comprising: a graphical representation analyzer configured to: receive, from the computing device, a video signal comprising a graphical representation of content to be rendered in a display region of an application window displayed via a display device coupled to the apparatus; analyze the graphical representation to determine characteristics of the display region of the application window, characteristics of the graphical representation, and a network address at which the content is located; retrieve the content based on the network address; and render the content over the display region of the application window in accordance with the characteristics determined for the display region of the application window and the characteristics determined for the graphical representation.

In an embodiment of the apparatus, the apparatus comprises a docking station.

In an embodiment of the apparatus, the graphical representation is a quick response (QR) code.

In an embodiment of the apparatus, the characteristics of the display region of the application window comprise at least one of: a first dimension value representative of a first dimension of the display region of the application window; or a second dimension value representative of a second dimension of the display region of the application window; and wherein the characteristics of the graphical representation comprise at least one of: an offset value representative of an offset of the graphical representation from a first origin coordinate of the display region of the application window; a third dimension value representative of a third dimension of the graphical representation; or a fourth dimension value representative of a fourth of the graphical representation.

In an embodiment of the apparatus, the apparatus is further configured to: determine at least one of a fifth dimension value representative of a fifth dimension of the graphical representation as displayed in the display region of the application window or a sixth dimension value representative of a sixth dimension of the graphical representation as displayed in the display region of the application window; determine a scaling factor based on at least one of the first dimension value or the second dimension value and at least one of the fifth dimension value or the sixth dimension value; scale the content in accordance with the scaling factor; and render the scaled content over the display region of the application window.

In an embodiment of the apparatus, the apparatus is further configured to: scale the first dimension value and the second dimension value of the display region of the application window in accordance with the scaling factor to determine dimensions of the content to be rendered over the display region of the application window; determine a second origin coordinate based on a coordinate at which the graphical representation is located with respect to the display device and the offset value; and scale the second origin coordinate to determine a location at which the scaled content is to be rendered over the display region of the application window.

In an embodiment of the apparatus, the apparatus is further configured to: detect a triggering event; and overlay the content on top of the graphical representation responsive to detecting the triggering event.

In an embodiment of the apparatus, the apparatus is further configured to: detect a cursor rendered by the computing device is placed within the display region of the application window.

A method performed by a device configured to be externally coupled to a computing device is also described herein. The method includes: receiving, from the computing device, a video signal comprising a graphical representation of content to be rendered in a display region of an application window displayed via a display device coupled to the device configured to be externally coupled to the computing device; analyzing the graphical representation to determine characteristics of the display region of the application window, characteristics of the graphical representation, and a network address at which the content is located; retrieving the content based on the network address; and rendering the content over the display region of the application window in accordance with the characteristics determined for the display region of the application window and the characteristics determined for the graphical representation.

In an embodiment of the foregoing method, the device configured to be externally coupled to the computing device comprises a docking station.

In an embodiment of the foregoing method, the device configured to be externally coupled to the computing device is incorporated into the display device.

In an embodiment of the foregoing method, the graphical representation is a quick response (QR) code.

In an embodiment of the foregoing method, the characteristics of the display region of the application window comprise at least one of: a first dimension value representative of a first dimension of the display region of the application window; or a second dimension value representative of a second dimension of the display region of the application window; and wherein the characteristics of the graphical representation comprise at least one of: an offset value representative of an offset of the graphical representation from a first origin coordinate of the display region of the application window; a third dimension value representative of a third dimension of the graphical representation; or a fourth dimension value representative of a fourth of the graphical representation.

In an embodiment of the foregoing method, rendering the content comprises: determining at least one of a fifth dimension value representative of a fifth dimension of the graphical representation as displayed in the display region of the application window or a sixth dimension value representative of a sixth dimension of the graphical representation as displayed in the display region of the application window; determining a scaling factor based on at least one of the first dimension value or the second dimension value and at least one of the fifth dimension value or the sixth dimension value; scaling the content in accordance with the scaling factor; and rendering the scaled content over the display region of the application window.

In an embodiment of the foregoing method, scaling the content comprises: scaling the first dimension value and the second dimension value of the display region of the application window in accordance with the scaling factor to determine dimensions of the content to be rendered over the display region of the application window; determining a second origin coordinate based on a coordinate at which the graphical representation is located with respect to the display device and the offset value; and scaling the second origin coordinate to determine a location at which the scaled content is to be rendered over the display region of the application window.

In an embodiment of the foregoing method, rendering the content comprises: detecting a triggering event; and overlaying the content on top of the graphical representation responsive to detecting the triggering event.

In an embodiment of the foregoing method, the triggering event comprises: detecting a cursor rendered by the computing device is placed within the display region of the application window.

A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor of a device configured to be externally coupled to a computing device, perform a method. The method includes: receiving, from the computing device, a video signal comprising a graphical representation of content to be rendered in a display region of an application window displayed via a display device coupled to the device configured to be externally coupled to the computing device; analyzing the graphical representation to determine characteristics of the display region of the application window, characteristics of the graphical representation, and a network address at which the content is located; retrieving the content based on the network address; and rendering the content over the display region of the application window in accordance with the characteristics determined for the display region of the application window and the characteristics determined for the graphical representation.

In an embodiment of the foregoing computer-readable storage medium, the graphical representation is a quick response (QR) code.

In an embodiment of the foregoing computer-readable storage medium, the characteristics of the display region of the application window comprise at least one of: a first dimension value representative of a first dimension of the display region of the application window; or a second dimension value representative of a second dimension of the display region of the application window; and wherein the characteristics of the graphical representation comprise at least one of: an offset value representative of an offset of the graphical representation from a first origin coordinate of the display region of the application window; a third dimension value representative of a third dimension of the graphical representation; or a fourth dimension value representative of a fourth of the graphical representation.

V. Conclusion

While various embodiments of the disclosed subject matter have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the relevant art(s) that various changes in form and details may be made therein without departing from the spirit and scope of the embodiments as defined in the appended claims. Accordingly, the breadth and scope of the disclosed subject matter should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.