VIRTUAL NETWORK COMPUTING WITH EXTRA KEYBOARD LAYOUT

Methods, systems, and techniques are provided for displaying objects in virtual network computing (VNC). For example, a VNC connection may be established between a first device and a second device, where the VNC connection enables a synchronization of an interactive display layout from the first device to the second device. Subsequently, after the VNC connection is established, a page structure of the first device may be retrieved based on an application programming interface (API) on the second device. In some embodiments, based on the retrieved page structure, one or more non-interactive objects on the second device may be displayed, where the one or more non-interactive objects are displayed on top of at least a portion of the interactive display layout at the second device.

FIELD OF THE DISCLOSURE

The present disclosure is generally directed to virtual network computing (VNC), in particular, toward displaying objects in a VNC system.

BACKGROUND

Virtual Network Computing (VNC) is a graphical device-sharing system that uses a Remote Frame Buffer (RFB) protocol to remotely control another device. For example, VNC transmits inputs (e.g., keyboard inputs, mouse inputs, gestures, taps, long-presses, swipes, etc.) from one device to another, relaying the graphical-screen updates, over a network. In some cases, VNC may be used for remote technical support, accessing files on a work computer from a home computer or vice versa, or otherwise accessing a remote device from a local device.

BRIEF SUMMARY

Example aspects of the present disclosure include:

A method for displaying objects in a VNC system, the method comprising: establishing a VNC connection between a first device and a second device, the VNC connection enabling a synchronization of an interactive display layout from the first device to the second device; retrieving, based at least in part on an application programming interface (API) on the second device, a page structure of the first device after the VNC connection is established; and displaying one or more non-interactive objects on the second device based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device.

Any of the aspects herein, wherein retrieving the page structure comprises: retrieving the page structure of the first device in an extensible markup language (XML) type.

Any of the aspects herein, wherein the XML type comprises specified properties for each of a plurality of objects displayed on the first device according to the page structure.

Any of the aspects herein, wherein the specified properties for each of the plurality of objects comprise an x-coordinate, a y-coordinate, a width, a height, an index, a name, or a combination thereof, for each of the plurality of objects.

Any of the aspects herein, wherein the plurality of objects comprises a plurality of buttons of an alpha-numerical keyboard layout.

Any of the aspects herein, wherein displaying the one or more non-interactive objects on top of at least the portion of the interactive display layout at the second device prevents a user of the second device to interact with elements of the interactive display layout at the second device that are beneath the one or more non-interactive objects.

Any of the aspects herein, wherein the one or more non-interactive objects comprise a semi-transparent display of an alpha-numerical keyboard layout, and wherein the alpha-numerical keyboard layout is displayed on the first device.

Any of the aspects herein, further comprising: detecting an interaction made by a user of the second device, the interaction corresponding to a location on the interactive display layout at the second device where the one or more non-interactive objects are being displayed; and processing the interaction on the interactive display layout at the first device.

A system for displaying objects in VNC, comprising: a processor; and a memory storing data for processing by the processor, the data, when processed, causes the processor to: establish a VNC connection between a first device and a second device, the VNC connection enabling a synchronization of an interactive display layout from the first device to the second device; retrieve, based at least in part on an API on the second device, a page structure of the first device after the VNC connection is established; and display one or more non-interactive objects on the second device based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device.

Any of the aspects herein, wherein the data stored in the memory that, when processed causes the processor to retrieve the page structure causes the system to: retrieve the page structure of the first device in an XML type.

Any of the aspects herein, wherein the XML type comprises specified properties for each of a plurality of objects displayed on the first device according to the page structure.

Any of the aspects herein, wherein the specified properties for each of the plurality of objects comprise an x-coordinate, a y-coordinate, a width, a height, an index, a name, or a combination thereof, for each of the plurality of objects.

Any of the aspects herein, wherein the plurality of objects comprises a plurality of buttons of an alpha-numerical keyboard layout.

Any of the aspects herein, wherein displaying the one or more non-interactive objects on top of at least the portion of the interactive display layout at the second device prevents a user of the second device to interact with elements of the interactive display layout at the second device that are beneath the one or more non-interactive objects.

Any of the aspects herein, wherein the one or more non-interactive objects comprise a semi-transparent display of an alpha-numerical keyboard layout, and wherein the alpha-numerical keyboard layout is displayed on the first device.

Any of the aspects herein, wherein the memory stores further data for processing by the processor that, when processed, causes the processor to: detect an interaction made by a user of the second device, the interaction corresponding to a location on the interactive display layout at the second device where the one or more non-interactive objects are being displayed; and process the interaction on the interactive display layout at the first device.

A system for displaying objects in VNC, comprising: means to establish a VNC connection between a first device and a second device, the VNC connection enabling a synchronization of an interactive display layout from the first device to the second device; means to retrieve, based at least in part on an API on the second device, a page structure of the first device after the VNC connection is established; and means to display one or more non-interactive objects on the second device based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device.

Any of the aspects herein, further comprising: means to retrieve the page structure of the first device in an XML type.

Any of the aspects herein, wherein the XML type comprises specified properties for each of a plurality of objects displayed on the first device according to the page structure.

Any of the aspects herein, wherein the specified properties for each of the plurality of objects comprise an x-coordinate, a y-coordinate, a width, a height, an index, a name, or a combination thereof, for each of the plurality of objects.

Any of the aspects herein, wherein the plurality of objects comprises a plurality of buttons of an alpha-numerical keyboard layout.

Any of the aspects herein, wherein displaying the one or more non-interactive objects on top of at least the portion of the interactive display layout at the second device prevents a user of the second device to interact with elements of the interactive display layout at the second device that are beneath the one or more non-interactive objects.

Any of the aspects herein, wherein the one or more non-interactive objects comprise a semi-transparent display of an alpha-numerical keyboard layout, and wherein the alpha-numerical keyboard layout is displayed on the first device.

Any of the aspects herein, further comprising: means to detect an interaction made by a user of the second device, the interaction corresponding to a location on the interactive display layout at the second device where the one or more non-interactive objects are being displayed; and means to process the interaction on the interactive display layout at the first device.

Any one or more of the features disclosed herein.

Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the embodiment descriptions provided hereinbelow.

DETAILED DESCRIPTION

In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or combinations thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media includes non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., random-access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

While machine-executable instructions may be stored and executed locally to a particular machine (e.g., personal computer, mobile computing device, laptop, etc.), it should be appreciated that the storage of data and/or instructions and/or the execution of at least a portion of the instructions may be provided via connectivity to a remote data storage and/or processing device or collection of devices, commonly known as “the cloud,” but may include a public, private, dedicated, shared and/or other service bureau, computing service, and/or “server farm.”

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Qualcomm® Snapdragon® 800 and 801, Qualcomm® Snapdragon® 610 and 615 with 4G LTE Integration and 64-bit computing, Apple® A7 microprocessor with 64-bit architecture, Apple® M7 motion comicroprocessors, Samsung® Exynos® series, the Intel® Core™ family of microprocessors, the Intel® Xeon® family of microprocessors, the Intel® Atom™ family of microprocessors, the Intel Itanium® family of microprocessors, Intel® Core i5-4670K and i-4770K 22 nm Haswell, Intel® Core® i5-3570K 22 nm Ivy Bridge, the AMD® FX™ family of microprocessors, AMD® FX-4300, FX-6300, and FX-8350 32 nm Vishera, AMD® Kaveri microprocessors, Texas Instruments® Jacinto C6000™ automotive infotainment microprocessors, Texas Instruments® OMAP™ automotive-grade mobile microprocessors, ARM® Cortex™-M microprocessors, ARM® Cortex-A and ARM926EJ-S™ microprocessors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia Geforce RTX 2000-series processors, Nvidia Geforce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements. The processors listed herein are not intended to be an exhaustive list of all possible processors that can be used for implementation of the described techniques, and any future iterations of such chips, technologies, or processors may be used to implement the techniques and embodiments of the present disclosure as described herein.

Various aspects of the present disclosure will be described herein with reference to drawings that may be schematic illustrations of idealized configurations.

As described previously, Virtual Network Computing (VNC) is a graphical device-sharing system that uses a Remote Frame Buffer (RFB) protocol to remotely control another device. For example, VNC transmits inputs (e.g., keyboard inputs, mouse inputs, gestures, taps, long-presses, swipes, etc.) from one device to another, relaying the graphical-screen updates, over a network. In some cases, VNC may be used for remote technical support, accessing files on a work computer from a home computer or vice versa, or otherwise accessing a remote device from a local device.

In some examples, VNC may be enabled in part by a VNC server that is a program on a machine (e.g., device) that shares some screen (and may not be related to a physical display, such that the server can be “headless”) and allows a client to share control of the screen or access the screen. Additionally, a VNC client (or viewer) is a program that represents the screen data originating from the VNC server, receives updates from the VNC server, and presumably controls the screen by informing the VNC server of collected local input. A VNC protocol (e.g., RFB protocol transmits one graphic primitive from the VNC server to the VNC client (e.g., “Put a rectangle of pixel data at a specified X, Y position”) and event messages from the VNC client to the VNC server.

In some embodiments, an object (e.g., a soft keyboard) may disappear on VNC when a user is interacting with a local device (e.g., virtual device) to view or perform actions on a remote device (e.g., real or actual device), such as inputting a password and passcode on the local device. Additionally, users may experience difficulty to provide inputs (e.g., text inputs) if the object (e.g., keyboard) is not displayed through the VNC connection, even though the keyboard is present or appears on the remote device. As an example, when a user attempts to perform a “Tap” action on a button on the local device (e.g., “Login” button) after the user inputted a password completely through the VNC connection (e.g., web VNC component), the user may be prevented from performing the “Tap” action as expected.

In some embodiments, the user may be prevented from performing the “Tap” action based on the object (e.g., soft keyboard layout) being blocked from device screenshots and screen recordings when users are providing inputs on the local device (e.g., inputting password or passcode on the local device). That is, buttons on the local device (e.g., “Login” button) may be covered by the blocked or unseen object (e.g., the soft keyboard layout is present on top of the buttons even though the layout is not shown), thereby preventing the user from interacting with the buttons (e.g., performing the “Tap” action). For example, the “Login” button may be non-interactive at that time based on the soft keyboard layout being present over the “Login” button on the remote or actual device, while the soft keyboard layout is blocked from being displayed on the local or virtual device.

It is with respect to the above issues and other problems that the embodiments presented herein were contemplated.

As described herein, an Application Programming Interface (API) is provided on a device which can retrieve the page structure of the device (e.g., in XML type). The API may allow users to do automation tests against their own applications on the device. Relying on the page structure and corresponding XML objects (e.g., with properties like “x,” “y,” “width,” “height,” “index,” “name,” etc.), the API can draw up each button from a keyboard (e.g., and/or other types of objects) and assemble and combinate the objects into a dummy semi-transparent keyboard web layout above the VNC component. In some embodiments, the dummy semi-transparent keyboard web layout may not respond to any actions from the end users. Additionally, operation instructions will be passed to the VNC component below, such that VNC interactions will not be blocked. Accordingly, the dummy semi-transparent keyboard web layout enables users to know a real state of the soft keyboard on the device and guide the users to perform operations through the layout.

The API and techniques provided herein may improve interaction with remote devices through a web VNC component at local devices for users. Additionally, the API and techniques may enable the users to have more accurate real-time user experiences when using the VNC component.

Referring now toFIG.1, a system100is shown in accordance with embodiments of the present disclosure. The system100may comprise a first device102, a second device104, and a processor106. In some embodiments, the first device102and the second device104may each represent one of a various types of user equipment, such as personal computers, mobile computing devices, smartphones, laptops, desktop, tablets, etc. The processor106may be any processor described herein or any similar electronic processor.

As described herein, the processor106may be configured to support a VNC component108that enables a VNC connection between the first device102and the second device104. For example, the VNC connection may be a graphical device-sharing connection that uses protocols (e.g., RFB protocol) to remotely control the first device102and/or the second device104from the opposite device or another device. The VNC component108may transmit inputs (e.g., keyboard inputs, mouse inputs, gestures, taps, long-presses, swipes, etc.) from one device to another, relaying the graphical-screen updates, over a network. In some examples, the VNC component108may be used for remote technical support (e.g., for an information technology (IT) specialist to remotely access a device for troubleshooting and/or performing various operations on the device), accessing files on a work computer from a home computer (or vice versa), or otherwise accessing a remote device from a local device.

In some cases, one or more objects may disappear on the VNC connection when a user is interacting with a local device to view or perform actions on a remote device. For example, the object(s) may be blocked from being displayed in device screenshots and screen recordings when users are providing inputs on the local device using the VNC connection.

Methods, systems, and techniques are provided herein for retrieving a page structure of the remote device (e.g., in XML type) and drawing objects on the local device based on the page structure. For example, the page structure may include properties for the object(s), such as coordinates (e.g., “x” coordinate, “y” coordinate, etc.), a width, a height, an index, a name, etc. Accordingly, the methods, systems, and techniques can draw, assemble, and combinate the object(s) into a dummy semi-transparent web layout above the VNC component108on the local device. In some embodiments, the dummy semi-transparent web layout may not respond to any actions from the end users. Additionally, operation instructions will be passed to the VNC component108below, such that VNC interactions will not be blocked. Accordingly, the dummy semi-transparent web layout enables users to know a real state of object(s) being displayed on the remote device and guide the users to perform operations through the layout.

FIG.2depicts an interactive VNC display200in accordance with at least one embodiment of the present disclosure. The interactive VNC display200may implement aspects of or may be implemented by aspects ofFIG.1. For example, the interactive VNC display200may be displayed on a local device (e.g., through a VNC component108) representing a graphical display of a remote device after a VNC connection is established between the local device and the remote device. Accordingly, the interactive VNC display200may enable a user at the local device to remotely control the remote device, such as providing a first input202(e.g., username) and a second input204(e.g., password or passcode). Additionally, the interactive VNC display200may be configured to enable the user at the local device to perform other actions, such as performing a “Tap” action on a button206(e.g., “Login” button).

However, in some examples, when trying to perform the “Tap” action on the button206after the first input202and the second input204are completely input (e.g., through a web VNC component), the user at the local device may be unable to perform the “Tap” action as expected. For example, a soft keyboard layout (e.g., and/or other objects) may be blocked from being displayed in device screenshots and screen recordings (e.g., captured through the VNC component) when users are providing the first input202and/or the second input204on the local device. That is, the button206may be covered by the soft keyboard layout, such that the button206is non-interactive at that time.

FIG.3depicts an interactive VNC display300with a keyboard layout302in accordance with at least one embodiment of the present disclosure. The interactive VNC display300may implement aspects of or may be implemented by aspects ofFIGS.1and2. For example, the interactive VNC display300may include the first input202, the second input204, and the button206as described with reference toFIG.2. Additionally, the interactive VNC display300may be displayed on a local device (e.g., through a VNC component108) representing a graphical display of a remote device after a VNC connection is established between the local device and the remote device. However, as described previously, the button206may be covered by a soft keyboard layout, such that the button206is non-interactive when users are providing the first input202and/or the second input204on the local device.

As described herein, an API on the local device is provided that can retrieve a page structure of the remote device in XML type. The API allows the users at the local device to perform automation tests against applications on the local device. Relying on the retrieved page structure and corresponding XML objects that include specific properties for each object (e.g., such as display coordinates, width, height, index, name, etc.), the VNC component and/or local device can draw up each button of the keyboard layout302and assemble and combinate all the objects into a dummy semi-transparent keyboard web layout above the VNC component. The semi-transparent keyboard web layout will not respond to any actions from the end users. Operation instructions will be passed to the VNC component below, such that VNC interactions will not be blocked. Additionally, the semi-transparent keyboard web layout can let users know the real state of the soft keyboard on the local device and guide them to perform operations through the layout.

Turning toFIG.4, a block diagram of a system400according to at least one embodiment of the present disclosure is shown. In one embodiment, system400is used to interconnect first device412and second device418for remote communications therebetween via a network, as described with reference toFIGS.1-3. The system400comprises a computing device402, a first device412, a second device418, a database430, and/or a cloud434or other network. Systems according to other embodiments of the present disclosure may comprise more or fewer components than the system400. For example, embodiments of the system400may omit the first device412, the second device418, one or more components of the computing device402, the database430, and/or the cloud434. Additionally or alternatively, the system400may include additional devices.

The computing device402comprises a processor404, a memory406, a communication interface408, and a user interface410. Computing devices according to other embodiments of the present disclosure may comprise more or fewer components than the computing device402. In some embodiments, the computing device402may be part of the first device412or the second device418. Additionally or alternatively, each of the first device412and the second device418may comprise a respective computing device for enabling or performing the techniques described herein to display a keyboard (e.g., or additional objects) for VNC.

The processor404of the computing device402may be any processor described herein or any similar electronic processor. For example, the processor404may be represented by the processor106as described with reference toFIG.1. The processor404may be configured to execute instructions or data stored in the memory406, which the instructions or data may cause the processor404to carry out one or more computing steps utilizing or based on data received from the first device412, the second device418, the database430, and/or the cloud434.

The memory406may be or comprise RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer-readable data and/or instructions. The memory406may store information or data useful for completing and/or means to perform, for example, any step of the methods500,600, and/or700described herein, or of any other methods. The memory406may store, for example, instructions that support one or more functions of the first device412and/or the second device418. For instance, the memory406may store content (e.g., instructions) that, when executed by the processor404, enable VNC connection establishment420, page structure retrieval422, object display424, and/or interaction processing428.

The VNC connection establishment420enables the processor404to establish a VNC connection between the first device412and the second device418, the VNC connection enabling a synchronization of an interactive display layout from the first device412to the second device418. For example,

The page structure retrieval422enables the processor404to retrieve, based at least in part on an API on the second device418, a page structure of the first device412after the VNC connection is established. For example, the page structure retrieval422enables the processor404to retrieve the page structure of the first device412in an XML type. In some examples, the XML type comprises specified properties for each of a plurality of objects displayed on the first device412according to the page structure. For example, the specified properties for each of the plurality of objects comprise an x-coordinate, a y-coordinate, a width, a height, an index, a name, or a combination thereof, for each of the plurality of objects. In some embodiments, the plurality of objects comprises a plurality of buttons of an alpha-numerical keyboard layout.

The object display424enables the processor404to display one or more non-interactive objects on the second device418based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device418. In some embodiments, displaying the one or more non-interactive objects on top of at least the portion of the interactive display layout at the second device418prevents a user of the second device418to interact with elements of the interactive display layout at the second device418that are beneath the one or more non-interactive objects. Additionally, in some embodiments, the one or more non-interactive objects comprise a semi-transparent display of an alpha-numerical keyboard layout, where the alpha-numerical keyboard layout is displayed on the first device412.

The interaction processing428enables the processor404to detect an interaction made by a user of the second device418, the interaction corresponding to a location on the interactive display layout at the second device418where the one or more non-interactive objects are being displayed. Subsequently, in some embodiments, the interaction processing428enables the processor404to process the interaction on the interactive display layout at the first device412.

Content stored in the memory406, if provided as in instruction, may, in some embodiments, be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, the memory406may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor404to carry out the various method and features described herein. Thus, although various contents of memory406may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and/or machine learning models. The data, algorithms, and/or instructions may cause the processor404to manipulate data stored in the memory406and/or received from or via the first device412, the second device418, the database430, and/or the cloud434.

The computing device402may also comprise a communication interface408. The communication interface408may be used for receiving image data or other information from an external source (such as the first device412, the second device418, the database430, the cloud434, and/or any other system or component not part of the system400), and/or for transmitting instructions, images, or other information to an external system or device (e.g., another computing device402, the first device412, the second device418, the database430, the cloud434, and/or any other system or component not part of the system400). The communication interface408may comprise one or more wired interfaces (e.g., a USB port, an Ethernet port, a Firewire port) and/or one or more wireless transceivers or interfaces (configured, for example, to transmit and/or receive information via one or more wireless communication protocols such as 802.11a/b/g/n, Bluetooth, NFC, ZigBee, and so forth). In some embodiments, the communication interface408may be useful for enabling the computing device402to communicate with one or more other processors404or computing devices402, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

The computing device402may also comprise one or more user interfaces410. The user interface410may be or comprise a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and/or any other device for receiving information from a user and/or for providing information to a user. The user interface410may be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Notwithstanding the foregoing, any required input for any step of any method described herein may be generated automatically by the system400(e.g., by the processor404or another component of the system400) or received by the system400from a source external to the system400. In some embodiments, the user interface410may be useful to enable a user (e.g., and/or technical support) to establish and facilitate a VNC connection between the first device412and the second device418based on the instructions to be executed by the processor404according to one or more embodiments of the present disclosure, and/or to modify or adjust a setting of other information displayed on the user interface410or corresponding thereto.

Although the user interface410is shown as part of the computing device402, in some embodiments, the computing device402may utilize a user interface410that is housed separately from one or more remaining components of the computing device402. In some embodiments, the user interface410may be located proximate one or more other components of the computing device402, while in other embodiments, the user interface410may be located remotely from one or more other components of the computer device402.

The database430may be configured to provide any such information to the computing device402or to any other device of the system400or external to the system400, whether directly or via the cloud434. In some embodiments, the database430may provide information for facilitating the VNC connection between the first device412and the second device418.

The cloud434may be or represent the Internet or any other wide area network. The computing device402may be connected to the cloud434via the communication interface408, using a wired connection, a wireless connection, or both. In some embodiments, the computing device402may communicate with the database430and/or an external device (e.g., a computing device) via the cloud434.

The system400or similar systems may be used, for example, to carry out one or more aspects of any of the methods500,600, and/or700described herein. The system400or similar systems may also be used for other purposes.

FIG.5depicts a method500that may be used, for example, to display objects in a VNC system and/or through a VNC connection.

The method500(and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s)404of the computing device402described above. The at least one processor may be part of a remote device and/or a local device. A processor other than any processor described herein may also be used to execute the method500. The at least one processor may perform the method500by executing elements stored in a memory such as the memory406. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method500. One or more portions of a method500may be performed by the processor executing any of the contents of memory, such as a VNC connection establishment420, a page structure retrieval422, an object display424, and/or an interaction processing428.

The method500comprises establishing a VNC connection between a first device and a second device, the VNC connection enabling a synchronization of an interactive display layout from the first device to the second device (step502). In some examples, the first device may be a remote device (e.g., actual device), and the second device may be a local device (e.g., device that accesses the remote device through a VNC component).

The method500also comprises retrieving, based at least in part on an API on the second device, a page structure of the first device after the VNC connection is established (step504). In some embodiments, the API may enable users to perform automation tests against applications on the first device (e.g., remote device).

The method500also comprises displaying one or more non-interactive objects on the second device based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device (step506). In some embodiments, displaying the one or more non-interactive objects on top of at least the portion of the interactive display layout at the second device may prevent a user of the second device to interact with elements of the interactive display layout at the second device that are beneath the one or more non-interactive objects. Additionally, the one or more non-interactive objects may comprise a semi-transparent display of an alpha-numerical keyboard layout, where the alpha-numerical keyboard layout is displayed on the first device.

The present disclosure encompasses embodiments of the method500that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above.

FIG.6depicts a method600that may be used, for example, to display objects in a VNC system and/or through a VNC connection based on properties of the objects retrieved through a page structure.

The method600(and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s)404of the computing device402described above. The at least one processor may be part of a remote device and/or a local device. A processor other than any processor described herein may also be used to execute the method600. The at least one processor may perform the method600by executing elements stored in a memory such as the memory406. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method600. One or more portions of a method600may be performed by the processor executing any of the contents of memory, such as a VNC connection establishment420, a page structure retrieval422, an object display424, and/or an interaction processing428.

The method600comprises establishing a VNC connection between a first device and a second device, the VNC connection enabling a synchronization of an interactive display layout from the first device to the second device (step602). The method600also comprises retrieving, based at least in part on an API on the second device, a page structure of the first device after the VNC connection is established (step604). Steps602and604may implement similar aspects of steps502and504as described with reference toFIG.5.

In some embodiments, the method600also comprises retrieving the page structure of the first device in an XML type (step606). Additionally, the XML type may comprise specified properties for each of a plurality of objects displayed on the first device according to the page structure. For example, the specified properties for each of the plurality of objects comprise an x-coordinate, a y-coordinate, a width, a height, an index, a name, or a combination thereof, for each of the plurality of objects. In some embodiments, the plurality of objects may comprise a plurality of buttons of an alpha-numerical keyboard layout.

The method600also comprises displaying one or more non-interactive objects on the second device based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device (step608). Step608may implement similar aspects of step506as described with reference toFIG.5.

The present disclosure encompasses embodiments of the method600that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above.

FIG.7depicts a method700that may be used, for example, to process an interaction performed in a VNC system and/or through a VNC connection.

The method700(and/or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s)404of the computing device402described above. The at least one processor may be part of a remote device and/or a local device. A processor other than any processor described herein may also be used to execute the method700. The at least one processor may perform the method700by executing elements stored in a memory such as the memory406. The elements stored in memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method700. One or more portions of a method700may be performed by the processor executing any of the contents of memory, such as a VNC connection establishment420, a page structure retrieval422, an object display424, and/or an interaction processing428.

The method700comprises establishing a VNC connection between a first device and a second device, the VNC connection enabling a synchronization of an interactive display layout from the first device to the second device (step702). The method700also comprises retrieving, based at least in part on an API on the second device, a page structure of the first device after the VNC connection is established (step704). The method700also comprises displaying one or more non-interactive objects on the second device based at least in part on the retrieved page structure, the one or more non-interactive objects being displayed on top of at least a portion of the interactive display layout at the second device (step706). Steps702,704, and706may implement aspects of steps502,504, and506as described with reference toFIG.5and/or steps602,604, and608as described with reference toFIG.6.

The method700also comprises detecting an interaction made by a user of the second device, the interaction corresponding to a location on the interactive display layout at the second device where the one or more non-interactive objects are being displayed (step708). The method700also comprises processing the interaction on the interactive display layout at the first device (step710).

The present disclosure encompasses embodiments of the method700that comprise more or fewer steps than those described above, and/or one or more steps that are different than the steps described above.

Additionally, as noted above, the present disclosure encompasses methods with fewer than all of the steps identified inFIGS.5,6, and7(and the corresponding description of the methods500,600, and700), as well as methods that include additional steps beyond those identified inFIGS.5,6, and7(and the corresponding description of the methods500,600, and700). The present disclosure also encompasses methods that comprise one or more steps from one method described herein, and one or more steps from another method described herein. Any correlation described herein may be or comprise a registration or any other correlation.

References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” “some embodiments,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in conjunction with one embodiment, it is submitted that the description of such feature, structure, or characteristic may apply to any other embodiment unless so stated and/or except as will be readily apparent to one skilled in the art from the description. The present disclosure, in various embodiments, configurations, and aspects, includes components, methods, processes, systems and/or apparatus substantially as depicted and described herein, including various embodiments, subcombinations, and subsets thereof. Those of skill in the art will understand how to make and use the systems and methods disclosed herein after understanding the present disclosure. The present disclosure, in various embodiments, configurations, and aspects, includes providing devices and processes in the absence of items not depicted and/or described herein or in various embodiments, configurations, or aspects hereof, including in the absence of such items as may have been used in previous devices or processes, e.g., for improving performance, achieving ease, and/or reducing cost of implementation.

It should be understood that every maximum numerical limitation given throughout this disclosure is deemed to include each and every lower numerical limitation as an alternative, as if such lower numerical limitations were expressly written herein. Every minimum numerical limitation given throughout this disclosure is deemed to include each and every higher numerical limitation as an alternative, as if such higher numerical limitations were expressly written herein. Every numerical range given throughout this disclosure is deemed to include each and every narrower numerical range that falls within such broader numerical range, as if such narrower numerical ranges were all expressly written herein.