Runtime text translation for virtual execution environments

Systems and methods for runtime text translation for virtual execution environments. An example method comprises: identifying, by a virtualization server application, a window associated with an application running in a virtual execution environment controlled by the virtualization server application; identifying, in metadata associated with the window, a first alphanumeric string to be displayed in the window, the first alphanumeric string comprising one or more words in a first natural language; translating the first alphanumeric string to a second natural language to produce a second alphanumeric string; modifying the metadata associated with the window to produce modified metadata including the second alphanumeric string; transmitting the modified metadata to a virtualization client application associated with the virtual execution environment; and causing the virtualization client application to render the window.

TECHNICAL FIELD

The present disclosure is generally related to virtualized computer systems, and is specifically related to systems and methods for runtime text translation for virtual execution environments.

BACKGROUND

Virtualization may be viewed as abstraction of hardware components into logical objects in order to allow a computer system to execute various software modules, for example, multiple operating systems, concurrently and in isolation from other software modules. Virtualization may be achieved by running a software layer, often referred to as a “virtual machine monitor” or “hypervisor,” above the hardware and below the virtual machines. A hypervisor may abstract the physical layer and present this abstraction to virtual machines to use, by providing interfaces between the underlying hardware and virtual devices of virtual machines. For example, processor virtualization may be implemented by the hypervisor scheduling time slots on one or more physical processors for a virtual machine, rather than a virtual machine actually having a dedicated physical processor.

DETAILED DESCRIPTION

Described herein are systems and methods for runtime text translation for virtual execution environments. In an illustrative example, a virtual execution environment may be implemented by a terminal session (also referred to as “remote desktop session”) running on a multi-user terminal server (also referred to as “remote desktop server”) in communication with a terminal service client application running on a client computer. In another illustrative example, a virtual execution environment may be implemented by a virtual machine (VM) running one or more guest applications under a guest operating system (OS) managed by a virtual machine monitor (VMM). The VMM may emulate the underlying hardware platform (e.g., the x86 platform), including emulating the processor, memory, and peripheral devices (such as network interface controllers, hard disk controllers, etc.).

Various software applications (e.g., entertainment, business, and/or communication applications), may fail to provide national language support, thus hindering their utilization by large segments of the user population. The systems and methods of the present disclosure allow on-the-fly (runtime) translation of texts that are being displayed by software applications running within virtual execution environments. Such texts may include alphanumeric strings that are displayed within the graphical user interface (GUI) controls (such as buttons, menus, drop-down lists, etc.), as well as other texts that are displayed within application windows. The target natural language for translating the alphanumeric strings may be chosen to match the language of the user interface of the client device or may be specified by the user.

In an illustrative example, the virtual execution environment may be implemented by a terminal session running on a terminal server in communication with a terminal service client application running on a client computer (e.g., a mobile computing device). A window manager running on the terminal server may iterate over the windows opened by one or more applications running within a user's terminal session. For each window, certain metadata may be identified, including, e.g., definitions of window elements (e.g., window title) and definitions of GUI controls that are displayed within the given window. The identified metadata may include alphanumeric strings (such as alphanumeric strings displayed within buttons, menus, drop-down lists, etc.), which may be presented in the natural language that may be different from the preferred natural language of the client device user. Therefore, the window manager may translate the alphanumeric strings into the target natural language (e.g., the natural language which is currently selected by the user interface of the client device). The translation may be performed, e.g., by looking up a translation cache associated with the terminal server and transmitting a request to a translation service responsive to detecting a cache miss. Responsive to receiving the translated alphanumeric strings, the window manager may modify the respective windows metadata. The modified metadata may then be transmitted to the client device for rendering the windows associated with the application, as described in more detail herein below.

In another illustrative example, the virtual execution environment may be executing code that has been originally developed for other platforms. The virtual execution environment may include one or more virtual machines managed by VMM, each of which may run one or more guest applications under control of the guest operating system. A window manager managed by the VMM may iterate over the windows opened by one or more applications running within a virtual machine. For each window, certain metadata may be identified, including, e.g., definitions of window elements (e.g., window title) and definitions of GUI controls that are displayed within the given window. The identified metadata may include alphanumeric strings (such as alphanumeric strings displayed within buttons, menus, drop-down lists, etc.), which may be presented in the natural language that may be different from the preferred natural language of the user of the client device. Therefore, the window manager may translate the alphanumeric strings into the target natural language (e.g., the natural language which is currently selected by the user interface of the client device). The translation may be performed, e.g., by looking up a translation cache associated with the VMM and transmitting a request to a translation service responsive to detecting a cache miss. Responsive to receiving the translated alphanumeric strings, the window manager may modify the respective windows metadata. The modified metadata may then be transmitted to the virtual machine for rendering the windows associated with the application on the client device, as described in more detail herein below.

In another illustrative example, runtime text translation may be performed by the virtualization client application (e.g., a terminal service client or a virtual machine client application). Responsive to receiving one or more graphical images to be rendered in a window associated with an application running in a virtual execution environment (e.g., a terminal session or a virtual machine), the virtualization client application may perform optical character recognition (OCR) of the images to produce one or more alphanumeric strings to be displayed in the window. The alphanumeric strings may then be translated to the target natural language and utilized for rendering the window.

The systems and methods described herein may be implemented by hardware (e.g., general purpose and/or specialized processing devices, and/or other devices and associated circuitry), software (e.g., instructions executable by a processing device), or a combination thereof. Various aspects of the above referenced methods and systems are described in details herein below by way of examples, rather than by way of limitation.

FIG. 1depicts a high-level diagram of an example computer system100implementing a terminal server managing one or more terminal sessions, in which the systems and methods described herein may be implemented. The computer system100may include one or more central processing units (CPUs, also referred to as “processors”)110, which may be communicatively coupled to one or more memory devices115and one or more input/output (I/O) devices110via a system bus125.

“Processor” herein refers to a device capable of executing instructions handling data in registers or memory, encoding arithmetic, logical, control flow, floating point or I/O operations. In one illustrative example, a processor may follow Von Neumann architectural model and may include an arithmetic logic unit (ALU), a control unit, and a plurality of registers. In a further aspect, a processor may be a single core processor which is typically capable of executing one instruction at a time (or process a single pipeline of instructions), or a multi-core processor which may simultaneously execute multiple instructions. In another aspect, a processor may be implemented as a single integrated circuit, two or more integrated circuits, or may be a component of a multi-chip module. A processor may also be referred to as a central processing unit (CPU). “Memory device” herein refers to a volatile or non-volatile memory, such as RAM, ROM, EEPROM, or any other device capable of storing data. “I/O device” herein refers to a device capable of inputting and/or outputting binary data. In an illustrative example, an I/O device may be provided by a network interface controller (NIC) or a block I/O device, such as a hard disk controller.

In accordance with one or more aspects of the present disclosure, the computer system100may implement a terminal server130for creating and managing one or more virtual execution environments represented by terminal sessions140which run in communication with respective client computer systems150. Each client computer system150may run a terminal service client application155that access, e.g., via the Remote Desktop Protocol (RDP) over one or more networks160, the associated terminal session140running on the terminal server130. Thus, the terminal session140provides the processor, memory, and disk resources of the virtual execution environment, while the terminal service client application155implements a seamless interface between the terminal session140and the display, keyboard, and other I/O devices (e.g., mouse, printers, and/or various USB peripheral devices) of the client computer150. In an illustrative example, a terminal session140may run one or more terminal session applications160, such as entertainment, business, and/or communication applications.

In certain implementations, the window manager180running on the terminal server130may iterate over the windows170opened by one or more applications160running within the terminal session140. For each window170, the window manager180may identify the respective window metadata including one or more alphanumeric strings, translate the alphanumeric string into the target natural language, and transmit the modified metadata including the translated alphanumeric strings to the client150for rendering the windows associated with the application, as described in more detail herein below with reference toFIG. 3.

Alternatively, the runtime text translation may be performed by the terminal service client155. Responsive to receiving from the terminal server130one or more graphical images to be rendered in a window associated with an application running in the terminal session140, the terminal service client155may perform optical character recognition (OCR) of the graphical images to produce one or more alphanumeric strings to be displayed in the window. The terminal service client155may then translate the alphanumeric strings to the target natural language and utilized the translated versions of the alphanumeric strings for rendering the window, as described in more detail herein below with reference toFIG. 4.

FIG. 2depicts a high-level diagram of an example computer system200running a VMM managing one or more virtual machines in which the systems and methods described herein may be implemented. The computer system200may include one or more CPUs220, which may be communicatively coupled to one or more memory devices215and one or more input/output (I/O) devices220via a system bus225.

In accordance with one or more aspects of the present disclosure, the computer system200may implement a virtual execution environment for executing code that may have been originally developed for other platforms. The virtual execution environment may include one or more virtual machines230, each of which may run a guest OS235managing one or more guest applications240, such as entertainment, business, and/or communication applications.

Execution of the virtual machine230may be facilitated by the VMM245that may emulate the underlying hardware platform (e.g., the x86 platform), including emulating the processor, memory, and peripheral devices (e.g., network interface controllers, hard disk controllers, etc.). In certain implementations, the VMM245may be executed in a separate privileged context, which is isolated from the kernel context of the host computer system200. Alternatively, the VMM245may be executed in the host kernel context or the host user space context. The VMM245may utilize various suitable APIs for privileged-level access to the host platform, including the processor220, memory225, and peripheral devices220(e.g., network interface controllers, hard disk controllers, etc.).

For improving efficiency and portability of the virtualization architecture, certain virtual machine functions may be performed by one or more instances of the VM client application255communicating with the virtual machine230via the VMM245. The VM client application255may run in a non-privileged execution mode (e.g., at the current privilege level (CPL) of 3) in the user space of the host computer system200. The VM client application255may utilize various host APIs for accessing host functions, e.g., system calls implementing I/O functions.

In certain implementations, the window manager180managed by VMM245may iterate over the windows260opened by one or more applications240running on the virtual machine230. For each window260, the window manager180may identify the respective window metadata including one or more alphanumeric strings, translate the alphanumeric string into the target natural language, and transmit the modified metadata including the translated alphanumeric strings to the virtual machine230for rendering the windows associated with the application, as described in more detail herein below with reference toFIG. 3.

Alternatively, the runtime text translation may be performed by the virtual machine client application255. Responsive to receiving from the terminal server130one or more graphical images to be rendered in a window associated with an application running in the virtual machine230, the virtual machine client application255may perform optical character recognition (OCR) of the graphical images to produce one or more alphanumeric strings to be displayed in the window. The virtual machine client application255may then translate the alphanumeric strings to the target natural language and utilized the translated versions of the alphanumeric strings for rendering the window, as described in more detail herein below with reference toFIG. 4.

FIG. 3depicts a flow diagram of an example method300of server-side runtime text translation for virtual execution environments, in accordance with one or more aspects of the present disclosure. In an illustrative example, a virtual execution environment may be implemented by a terminal session running on a terminal server in communication with a terminal service client application running on a client computer. Therefore, the method300and/or each of its individual functions, routines, subroutines, or operations may be performed by the window manager180running under the terminal server130ofFIG. 1.

In another illustrative example, a virtual execution environment may be implemented by a virtual machine running one or more guest applications under a guest OS managed by a VMM. Therefore, the method300and/or each of its individual functions, routines, subroutines, or operations may be performed by the window manager280managed by the VMM243ofFIG. 2.

In certain implementations, the method300may be performed by a single processing thread. Alternatively, the method300may be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing the method300may be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method300may be executed asynchronously with respect to each other.

At block310, a window manager executed by a virtualization server application (represented, e.g., by the terminal server130ofFIG. 1or the VMM245ofFIG. 2) may identify the target natural language for translating the texts being displayed by software applications running in a virtual execution environment controlled by the virtualization server application (e.g., the terminal session140ofFIG. 1or the virtual machine230ofFIG. 2). In an illustrative example, the target natural language for translating the application texts may be specified by the virtualization client (e.g., the terminal service client155ofFIG. 1or the virtual machine230ofFIG. 2), e.g., to match the language of the user interface of the client device. In another illustrative example, the target natural language for translating the application texts may be specified by the user of the client device.

At block320, the window manager may identify one or more windows associated with an application running in the virtual execution environment. In an illustrative example, responsive to detecting a new virtual execution environment having been launched, the window manager may subscribe for notifications of new windows being created within the virtual execution environment. In another illustrative example, the window manager may subscribe for notifications of a window gaining the focus within the virtual execution environment. In yet another illustrative example, the window manager may traverse the list of processes running within the virtual execution environment and identify one or more windows associated with each running process. In yet another illustrative example, responsive to detecting a new application having been launched in the virtual execution environment, the window manager may traverse the list of windows associated with the application.

At block330, the window manager may identify the metadata associated with each identified window. The metadata may include, e.g., definitions of window elements (e.g., window title) and definitions of GUI controls that are displayed within the given window. In various illustrative examples, the window manager may identify the window metadata by executing certain system calls and/or inspecting system data structures associated with the identified windows.

At block340, the window manager may identify, within the window metadata, one or more alphanumeric strings to be displayed in the window. Each alphanumeric string may include one or more natural language words.

At block350, the window manager may translate the identified alphanumeric strings to the target natural language. In an illustrative example, the window manager may look up the alphanumeric strings in a translation cache associated with the virtualization server application. The translation cache may be provided by a data structure comprising a plurality of records, such that each record may map an alphanumeric string comprising one or more natural language words to a translated version of the alphanumeric string and an identifier of the target natural language. Responsive to failing to identify the alphanumeric strings to be translated in the translation cache, the window manager may transmit a translation request to an external translation service. The translation request may include one or more alphanumeric strings to be translated and may identify the target natural language for translation. Responsive to receiving the translation response, the window manager may create, for each translated alphanumeric string, a new record in the translation cache to store a mapping of the alphanumeric string to its translated version and an identifier of the target natural language.

In certain implementations, the window manager may post-process one or more translated alphanumeric strings, e.g., by truncating or otherwise abbreviating one or more words comprised by an alphanumeric string such that the size of the translated version of the alphanumeric string would not exceed the size of the original alphanumeric string.

At block360, the window manager may modify the window metadata by inserting the translated alphanumeric strings. In an illustrative example, the window manager may replace the original alphanumeric strings with their respective translated versions, so that the translated text would only be displayed within the windows rendered by the virtualization client application (e.g., the terminal service client155ofFIG. 1or the virtual machine client application255ofFIG. 2). In another illustrative example, the window manager may append the translated alphanumeric string to the window metadata, so that the virtualization client application would display both the original and the translated alphanumeric strings.

At block370, the window manager may transmit the modified metadata to the virtualization client application for rendering the respective windows.

FIG. 4depicts a flow diagram of an example method400of client-side runtime text translation for virtual execution environments, in accordance with one or more aspects of the present disclosure. In an illustrative example, a virtual execution environment may be implemented by a terminal session running on a terminal server in communication with a terminal service client application running on a client computer. Therefore, the method400and/or each of its individual functions, routines, subroutines, or operations may be performed by the terminal service client155in communication with terminal server140ofFIG. 1.

In another illustrative example, a virtual execution environment may be implemented by a virtual machine running one or more guest applications under a guest OS managed by a VMM. Therefore, the method400and/or each of its individual functions, routines, subroutines, or operations may be performed by the virtual machine client application255in communication with VMM245ofFIG. 2.

In certain implementations, the method400may be performed by a single processing thread. Alternatively, the method400may be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing the method400may be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method400may be executed asynchronously with respect to each other.

At block410, the virtualization client application implementing the method may identify the target natural language for translating the texts being displayed by software applications. In an illustrative example, the target natural language for translating the application texts may be selected to match the language of the user interface of the client device. In another illustrative example, the target natural language for translating the application texts may be specified by the user of the client device.

At block420, the virtualization client application may receive one or more graphical images to be rendered in one or more windows associated with an application running in a virtual execution environment managed by a virtualization server application. In an illustrative example, responsive to detecting a new application having been launched in the virtual execution environment, the virtualization client application may traverse the list of windows associated with the application and identify graphical images to be rendered within the respective windows.

At block430, the virtualization client application may perform an optical character recognition of at least part of each graphical image, thus producing one or more alphanumeric strings corresponding to the respective graphical images. Each alphanumeric string may include one or more natural language words.

At block440, the virtualization client application may translate each alphanumeric string to the target natural language. In an illustrative example, the virtualization client application may look up each alphanumeric string in a translation cache. The translation cache may be provided by a data structure comprising a plurality of records, such that each record may map an alphanumeric string comprising one or more natural language words to a translated version of the alphanumeric string and an identifier of the target natural language. Responsive to failing to identify the alphanumeric strings to be translated in the translation cache, the virtualization client application may transmit a translation request to an external translation service. The translation request may include one or more alphanumeric strings to be translated and may identify the target natural language for translation. Responsive to receiving the translation response, the virtualization client application may create, for each translated alphanumeric string, a new record in the translation cache to store a mapping of the alphanumeric string to its translated version and an identifier of the target natural language. In certain implementations, the virtualization client application may post-process one or more translated alphanumeric strings, e.g., by truncating or otherwise abbreviating one or more words comprised by an alphanumeric string such that the size of the translated version of the alphanumeric string would not exceed the size of the original alphanumeric string.

At block450, the virtualization client application may render the window comprising the translated alphanumeric strings. In an illustrative example, the original alphanumeric strings may be replaced with their respective translated versions, so that the translated text would only be displayed within the windows rendered by the virtualization client application. In another illustrative example, the virtualization client application may display both the original and the translated alphanumeric strings.

FIG. 5schematically illustrates a component diagram of an example computer system1000which may perform any one or more of the methods described herein. In various illustrative examples, the computer system1000may represent the example computer system100ofFIG. 1or example computer system200ofFIG. 2.

The example computer system1000may be connected to other computer systems in a LAN, an intranet, an extranet, and/or the Internet. The computer system1000may operate in the capacity of a server in a client-server network environment. The computer system1000may be a personal computer (PC), a set-top box (STB), a server, a network router, switch or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Further, while only a single example computer system is illustrated, the term “computer” shall also be taken to include any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

The computer system1000may comprise a processing device1002(also referred to as a processor or CPU), a main memory1004(e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory1006(e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device1018), which may communicate with each other via a bus1030.

The processing device1002represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device1002may be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processing device1002may also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. In accordance with one or more aspects of the present disclosure, the processing device1002may be configured to execute instructions implementing the methods300,400for runtime text translation for virtual execution environments.

The computer system1000may further comprise a network interface device1008, which may be communicatively coupled to a network1020. The computer system1000may further comprise a video display1010(e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device1012(e.g., a keyboard), a cursor control device1014(e.g., a mouse), and an acoustic signal generation device1016(e.g., a speaker).

The data storage device1018may include a computer-readable storage medium (or more specifically a non-transitory computer-readable storage medium)1028on which is stored one or more sets of executable instructions1026. In accordance with one or more aspects of the present disclosure, the executable instructions1026may comprise executable instructions encoding various functions of methods300,400for runtime text translation for virtual execution environments.

The executable instructions1026may also reside, completely or at least partially, within the main memory1004and/or within the processing device1002during execution thereof by the computer system1000, the main memory1004and the processing device1002also constituting computer-readable storage media. The executable instructions1026may further be transmitted or received over a network via the network interface device1008.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “identifying,” “determining,” “storing,” “adjusting,” “causing,” “returning,” “comparing,” “creating,” “stopping,” “loading,” “copying,” “throwing,” “replacing,” “performing,” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

The methods and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description below. In addition, the scope of the present disclosure is not limited to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present disclosure.