Patent ID: 12204555

DETAILED DESCRIPTION

Referring toFIG.1, a non-exclusive example embodiment of a system100includes a server computer102communicatively connected to a communications network104. The server computer102processes an application unit105of or accessible to the server computer102. The server computer102includes one or more computer system including a processor106, memory108, and a system bus110that couples system components, including the memory108, to the processor106. The memory108may include a read only memory (ROM)112and a random access memory (RAM)114. A basic input/output system (BIOS)116containing the basic routines that help to transfer information between elements within the computer system is stored in the ROM112. The server computer102also includes a storage drive118, and may also include other readable and writable storage120. The server computer102may also include input peripheral device121and output peripheral device122. The storage drive118, the other storage120, and the peripheral devices121,122are connected to the system bus110by relevant interface. A number of modules can be stored in the memory108, storage drive118or other storage120, including an operating system122and the application unit105. The server computer102also includes a communication interface device124for receiving and sending information over the communications network104.

Although the server computer102is illustrated as a single device, the server computer102could be a distributed computing system comprised of more than one server or computing device. The server computer102may, for example, be a cloud server.

A display device126is communicatively connected to the communications network104. The display device126provides inputs communicated over the communications network104to the server computer102for operations of the application unit105, and receives outputs of the application unit105and tags107from the server computer102. The tags107are indicia of the inputs received by the server computer102from the display device126for processing the application unit105. The display device126includes at least a processor128, memory130, input device132, and output device134. A system bus135connects the memory130, as well as the input device132and output device134, to the processor128. The display device126also includes a communication interface device131for sending and receiving information over the communications network104. The memory130includes a transform module136for operating on the tags107received from the server computer102.

The display device126may have limited or other processing capability. The display device126communicates over the network104with the server computer102. The server computer102provides processing of the application unit105accessed for input by and output to the display device126. The server computer102receives inputs to the display device126communicated over the network104, and sends processed data from the application unit105and tags107to the display device126for output. The display device126receives the tags107from the server computer102, together with processed data from the application unit105, and the transform module136operates with the tags107to affect the processed data for output on the display device126.

The communications network104includes any wired or wireless telecommunications system by which networked devices, such as the server computer102and the display device126, can exchange data. For example, the network104can include a local area network (LAN), a wide area network (WAN), an intranet, an Internet, or any combination.

An example embodiment of the system100includes a 3D rendering engine as the application unit. The display device126inputs a directive for communication over the network104to the server computer102. That input is received by the server computer102. The server computer102processes the 3D rendering engine according to the input. The server computer102then returns a framebuffer as processed data to the display device126together with tags107associated with the inputs that were received by the server computer102. The transform module136of the display device102operates on the tags107by comparing them to current inputs to the display device. The transform module136operates with the tags107to affect the framebuffer received for output by the display device126.

Referring toFIG.2, a system200includes a server computer201, such as the server computer102, communicating with a user device209, such as the display device126. The user device209sends input207directives from the user device209to the server computer201. The server computer201processes an application202, such as the application unit105, based on the input207. The server computer201returns to the user device209a processed data204, for example a processed framebuffer or other processed data, together with tags208indicative of the input207to the application202. A transformer205, such as the transform module136, of the user device209compares the tags208to any more current input to the user device209, and transforms the processed data204received for display or other output by the user device209to updated data206. The updated data206is output by the user device209.

In effect, the server computer201receives input207from the user device209, processes the input207and sends back processed data204. Bundled with the processed data204are tags208which specify values of input207that the application202processed to create the processed data204.

The user device209receives the processed data204and tags208and sends these to the transformer205. The transformer205uses the difference in the tags208and any more recent input to the user device209to operate on the processed data204. The result of the operation is updated data206. The updated data206reflects the current state of input to the user device209. The transformer205continues to create the updated data206with changes of input to the user device209, without any actions of the server computer201sending further processed data204from the application202.

Although single streams of processed data204and tags208are illustrated, it should be understood that multiple streams are possible.

Referring toFIG.3, a method300of operation of a server computer, such as the server computer102or201, includes receiving input302from a user device. The server computer receives the input and processes304an application, such as an application program, based on the input. The server computer then sends306processed data of the application to a user device over a network, together with tags indicative of the input on which the processed data is obtained.

Referring toFIG.4, a method400of operation of a user device, such as the display device126or user device209, includes input402to the user device communicated over the network to a server computer. The server computer processes an application with the input, and returns to the user device processed data and tags indicative of the input on which the processed data is obtained. The user device receives404the processed data and tags. The user device transforms406the processed data based on comparison of the tags to any more current input to the user device. The user device408outputs transformed data408on the user device.

Referring toFIG.5, an example embodiment of the foregoing includes a video rendering engine as the application of a server computer501accessed by a display device509. The video rendering engine includes a framebuffer510of the server computer. The application of the server computer sends framebuffer data503for display on an LCD504of the user device509. The application of the server computer501has created the framebuffer510with dimensions of 160×240, even though the LCD504has size519of 100×200. The larger framebuffer510is processed by a transformer in the user device509to adjust the area which is actually displayed in the LCD504.

Further, the application based on the last received input from the user device509fills the framebuffer510with data where the actual LCD504display area507is at offset 20,20. If no further inputs to the user device have occurred, the area507would be displayed on the LCD504at offset 20,20.

If, however, further inputs to the user device have occurred, the framebuffer data503is sent to the user device509and is placed in framebuffer508of the user device509. Framebuffer508has the area507at offset 20,20 that the application102determined should be placed on the LCD304based on the input that had been received. However, the new input to the user device has changed and the transformer of the user device509uses the new input and the tags to determine that area516at offset 25,30 should instead be displayed on the LCD. Area516is sent to the LCD for display on the user device509.

Referring toFIG.6, a system600includes a server computer602communicatively connected to a display604. The server computer602includes an application unit606. The display604communicates with the server computer602providing inputs for processing by the application unit604.

The application unit604running on the server computer602receives inputs607and data612from the display604. The application unit604processes the inputs607and data612, and sends back to the display604processed data617and tags618. The tags618specify values of inputs607that were received by the server computer602from the display604for processing of the processed data.

The display604has data606to send to the application unit604. Before the data606is sent to the server computer602by the display604, the data606is operated on by a transformer605of the display604. The transformer605transforms the data606to transformed data614based on differences between new input to the display604compared to the tags618received from the server computer602. The transformed data614is sent to the server computer602for processing by the application unit604. Together with the transformed data614, inputs that created the data614are sent by the display604to the server computer602. The inputs sent by the display604specify values of the new inputs used to create the transformed data.

Referring toFIG.7, a method700of operation of a server computer, such as the server computer602, includes receiving input702from a user device, such as the display604. The server computer receives the input and processes704an application, such as an application program, based on the input. The server computer then sends706processed data of the application to a user device over a network, together with tags indicative of the input on which the processed data is obtained. The user device has data to send to the server computer, together with inputs employed to create the data. The server computer receives708the data and inputs.

Referring toFIG.8, a method800of operation of a user device, such as the display604, includes input802to the user device communicated over the network to a server computer. The server computer processes an application with the input, and returns to the user device processed data and tags indicative of the input on which the processed data is obtained. The user device receives804the processed data and tags. The user device has data to send to the server computer. The user device transforms806the data to transformed data for sending to the server computer based on comparison of the tags to any more current input to the user device. The user device sends808the transformed data to the server computer, together with identifiers specifying the values of the more current input.

Referring toFIG.9, an example embodiment of the foregoing includes a display device909with a camera906. The camera906has created a framebuffer904which will be sent914to the application of a server computer. Based on inputs received by the application, the application of the server computer has sent data to the display. The data specifies two regions of interest, ROTA908and ROIB902. The application has bundled tags with the data. The tags indicate the input that was used by the application to create the regions of interest, ROTA908and ROIA902. The application has placed ROIA908at position 20,10 and ROIB902at position 200,40.

The display904receives the data and tags from the application of the server computer. The display904includes a transformer that compares more current inputs to the display904to the tags from the server computer. The transformer updates the region of interest positions. For example, ROIA907is positioned at 30,30 and ROIB903is positioned at 230,60. The transformer sets a mask to black out all areas except ROIA907and ROIB903. The specifying of the regions of interest and blacking out the rest of the framebuffer greatly reduces the bandwidth to send the framebuffer to the application of the server computer.

Referring toFIG.10, in conjunction withFIGS.2and6, an example embodiment of the foregoing includes a server computer1001processing a location determination module1011and an object creation module1015as an application. A display device1009with a camera1010, sensor1019, and LCD1004communicatively connects to the application of the server computer1001. The display device1009includes a first transformer1006and a second transformer1005.

The location determination module1011receives input1007from the sensor1019and data1014from the camera1010in the display device1009. Based on the camera data1014and sensor input1007, the location determination module1011determines the pixel location1013to place a 3D object on the LCD1004. The location determination module1011also sends data1017and associated tags1018to the display device1009for operations of the first transformer1006.

The object creation module1015uses the pixel location1013to create an object framebuffer data1014. The object creation module1015also sends to the display device1009tags1028containing the pixel location1013that was used to create the object framebuffer data1014.

The display device1009processes the first transformer1006using the data1017and associated tags1018sent from the location determination module1011together with current sensor input1007to create a local 3D pixel location1022. The display device1009processes the second transformer1005using the local 3D pixel location1022, the object framebuffer data1014and the tags1028which contain the server 3D pixel location to create an updated framebuffer data1016. The updated framebuffer data1016is used by the display device1009to draw an object1017on the LCD1004.

Referring toFIG.11, operations of a transformer, such as that of the example embodiment ofFIG.5, determine X Offset1105and Y Offset1106of an area to be displayed on an LCD of a display device based on mouse inputs. Constants1101are sent from the processed application of a server computer to the display device. Values1102are dimensions of the framebuffer and LCD of the display device which are known to the display device. Tags1103are sent by the server computer to the display device indicative of inputs received by the server computer for processing. Current mouse inputs1105are known values of the display device.

The X Offset1105computes the X offset of the area, making sure the area is always visible. The Y Offset1106computes the Y offset of the area, making sure the area is always visible. The X offset and Y offset are computed when either new mouse inputs occur or when new data is sent by the application of the server computer to the display device.

Referring toFIG.12, operations of a transformer, such as that of the example embodiment ofFIG.9, determine offsets for a region of interest to be displayed on an LCD of a display device based on mouse inputs. Operations of the transformer would occur for each region of interest. Constants1201are sent from the processed application of a server computer to the display device. Values1202are tags sent from the processed application to the display device indicative of inputs processed by the processed application. Values1203are the current input. Region of interest offsets1204are computed when either new mouse inputs occur or when new data is sent by the application of the server computer to the display device.

Referring toFIG.13, operations of a first transformer, such as that of the example embodiment ofFIG.10, determine the 3D position of an object to be displayed on an LCD of a display device based on sensor inputs of the display device. Constants1301are sent from the processed application of a server computer to the display device. Values1302are tags sent from the processed application to the display device indicative of inputs processed by the processed application. Value1303are the current sensor input. Updated X, Y, and Z positions1304for the 3D position of the object for display are computed when either new camera or sensor input occurs on the display device or new data is sent by the application of the server computer to the display device.

Operations of a second transformer, such as that of the example ofFIG.10, are similar to the operations of the transformer illustrated inFIG.11.

Referring toFIG.14, a timeline for operations of the transformer ofFIG.2, includes input1407D in the display is sent at time T1to the server and arrives some time later at time T2as input1407S. The application in the server processes that input1407S at time T3and generates data1404S and tags1408S at time T4. Input to the display occurs after the input1407D, for example, at time T5. Data1404S and tags1408S are sent by the server to the display at time T4, and arrive some time later at time T6at the display as data1404D and tags1408D. The display transforms1405D the data1404D at time T7using tags1408D and local input to the display, creating data1406D. The transformed data is output by the display at time T8.

The timeline continues for operations of the transformer with each new input1407D in the display.

In the foregoing specification, the invention has been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems and device(s), connection(s) and element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or element of any or all the claims. As used herein, the terms “comprises, “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.