Method and apparatus for controlling content on remote screen

There is provided a mobile terminal with a camera, such as a smart phone, capable of controlling the functions of a remote screen and content displayed on the remote screen. The mobile terminal defines a remote screen area in an image obtained by capturing the remote screen, receives remote screen resource information, and performs screen mapping between the remote screen area and the remote screen using the remote screen resource information. If it receives a user input signal related to the remote screen area, the mobile terminal generates event information for controlling an object based on the results of the screen mapping, and transmits the event information to a remote apparatus with the remote screen. The remote apparatus processes the event information and controls the object.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 U.S.C. §119(a) of Korean Patent Application No. 10-2011-0115912, filed on Nov. 8, 2011, the entire disclosure of which is incorporated herein by reference for all purposes.

BACKGROUND

The following description relates to an apparatus for remotely controlling content on the screen of a device, such as a TV, a PC, a notebook, etc., using a smart phone.

2. Description of the Related Art

In general, there are different kinds of devices such as a PC, a mobile phone, a TV, etc. They provide different services, such as web data search, call services, multimedia services, etc., depending on their intrinsic functions and content. However, due to the development of a smart phone and a smart TV, various kinds of content that were provided only by desktop computers are able to be consumed through smart mobile devices. In particular, a smart phone allows a user to receive his or her desired service or to do his or her works anytime, anywhere since the user can always carry the smart phone. Also, smart devices provide users with various kinds of services, such as music, video, news, etc., and connect to other smart devices through a network to offer interactive services.

Recently, as desktop computers, smart TVs, and smart phones can handle various kinds of content, content sharing is required gradually and also especially has drawn attention to home network and Digital Living Network Alliance (DLNA) technology. However, the technologies are focused on sharing of multimedia content such as music, images, video, or the like. The reason is that multimedia content is preferred content that can be controlled through a simple user interface. However, as a web browser is installed in many smart devices, multimedia, additional information (description, advertisement, games, a messenger, etc.), as well as applications allow users to be used. Therefore, a diversity of web content that can be displayed on the screen of a smart device is increasing. Accordingly, in order to select desired content among various kinds of web content, and share and move the selected content, there is substantial demand for improvement of a user interface.

The main differences of the iPhone and android phones from conventional smart phones are that the iPhone and android phones have an App store and a user interface allowing users to easily and conveniently use content in their mobile phones. In particular, a multi-touch interface allows users to easily zoom in/out or move content of a mobile phone. Recently, researches in technology for easily using three-Dimensional (3D) content-1 based on recognition of multi-touch gestures are also being conducted.

Generally, the multi-touch interface has been applied to a smart phone, a smart pad, and a notebook. Applying the multi-touch interface to an electronic device with a large screen, such as a smart TV, a projector, an electronic display board, etc., is not economical due to high manufacturing costs. In many cases, since such an electronic device with a large screen is remotely controlled, a simple button-type interface such as a remote controller has been used to control the functions of the electronic device. However, as a smart TV allows a web browser to display more various kinds of content on its screen and control more various applications, an integrated remote controller for dealing with such content and applications is needed. Furthermore, in the conventional remote controller, a functional extension of the remote controller was inevitably accompanied by addition of buttons or combinations of button inputs. That is, it will be difficult for a user to handle the controller when functions are added and the structure of the controller is complex. In order to overcome this problem, various technologies, such as an apparatus for recognizing a user's gestures using a camera, an apparatus for location detection using a laser pointer, a method of adding a mouse pad to a remote controller, etc., have been developed. However, these technologies are also quite costly.

An example of such conventional technology is disclosed in Korean Laid-open Patent Application No. 10-2006-0125735, entitled “Universal Remote Control Device with Touch Screen”, and laid-open on Dec. 6, 2006.

SUMMARY

The following description relates to a method of controlling a remote screen and content on the remote screen, using a smart phone that is always carried by a user and includes various input and output devices, such as buttons, a multi-touch screen, a voice recognition device, a camera, etc.

In one general aspect, there is provided a mobile terminal including: a photographing unit configured to capture a remote screen displaying content including one or more objects that are able to be controlled; a display unit configured to display an image of the captured remote screen; a user input unit configured to receive user input events for controlling at least one of the objects; a communication unit configured to receive remote screen resource information for controlling the objects on the remote screen, from a remote apparatus with the remote screen; and a controller configured to define a remote screen area in the image of the captured remote screen, to perform screen mapping between the rectangle bounds of remote screen area and the real remote screen, to generate, if user input events related to the rectangle of the remote screen are occurred, event information for controlling an object corresponding to the user input signal based on the results of the screen mapping, and to transmit the event information to the remote apparatus through the communication unit.

In another general aspect, there is provided a method in which a mobile terminal controls a remote screen, including: connecting to the remote screen through a network; receiving remote screen resource information from a remote apparatus with the remote screen, wherein the remote screen resource information is used to remotely control content including one or more controllable objects that are provided on the remote screen; defining a remote screen area in a remote screen image obtained by capturing the remote screen; performing screen mapping between the remote screen area and the remote screen, based on the remote screen resource information; receiving a user input signal related to the remote screen area; generating event information for controlling an object corresponding to the user input signal based on the results of the screen mapping; and transmitting the event information to the remote apparatus with the remote screen.

In another general aspect, there is provided a remote apparatus including: a display unit configured to provide content including one or more objects that are able to be controlled; a communication connection manager configured to receive input event information for controlling at least one of the objects, from a mobile terminal; a resource manager configured to control, if a communication connection request is received from the mobile terminal, the communication connection manager to provide the mobile terminal with remote screen resource information that is used to control the objects; an input event converter configured to convert the input event information of a mobile terminal into remote screen event information that is able to be used in a remote screen; a screen object information manager configured to manage representation information of the objects, and to select an object indicated by the remote screen event information, based on the representation information of the objects; and an input event processor configured to generate a control event for controlling the selected object.

In another general aspect, there is provided an operation method of a remote apparatus that is controlled by a mobile terminal, including: displaying content including one or more objects that are able to be controlled; receiving input event information for controlling the objects, from the mobile terminal; converting the input event information into remote screen event information that is used in a remote screen; selecting an object indicated by the remote screen event information based on representation information of the objects; and controlling the selected object using the remote screen event information.

DETAILED DESCRIPTION

FIG. 1shows an example of a system configuration environment for controlling a remote apparatus110using a mobile terminal120.

Referring toFIG. 1, the remote apparatus110includes a remote screen111for displaying content. The remote apparatus110has a user input and output interface that receives signals for controlling the remote apparatus110from a user input unit, such as a keyboard, a mouse, a touch screen, a trackball, etc. Also, the remote apparatus110may include a wired or wireless network device for connecting to external devices.

The mobile terminal120controls content of the remote apparatus110. The remote apparatus110may include all apparatuses, such as a desktop PC, a smart TV, a projector, a digital signage, a public display, any device with large screen, etc., which can provide visual information to users. The mobile terminal120may be an arbitrary terminal, such as a mobile phone, a smart phone, any device with a camera and a network adapter, etc., having a convenient input and output interface such as a touch screen. Also, the mobile terminal120may be configured to capture the remote screen111using a camera, and to enable a user to view content displayed on the remote screen111through a display screen121of the mobile terminal120. The content of the remote screen111, which is displayed on the display screen121, may be provided using a camera application.

According to an example, the mobile terminal120may operate in a remote control mode in which the content of the remote screen111is controlled using an image obtained by capturing the remote apparatus110. In the remote control mode, the user may perform a remote control function of selecting, moving, zooming in/out, and executing the content of the remote screen111through the display screen121of the mobile terminal120, using the user interface of the mobile terminal120, instead of the user interface of the remote screen111. The content includes one or more objects that can be controlled, and each object may be an arbitrary object, such as a mouse cursor, a user input button, link information, etc., which can be controlled according to an event generated by a user input signal.

FIG. 2is a flowchart illustrating an example of a process of transmitting a user control signal to the remote apparatus110through a camera screen of the mobile terminal120ofFIG. 1.

Referring toFIGS. 1 and 2, the mobile terminal120connects to the remote apparatus110with the remote screen111through a wired/wireless network.

The mobile terminal120receives remote screen resource information, such as a screen size, a connection session ID, a location, model information, etc., required for remote control, from the remote apparatus110. The remote screen resource information may include information about the thickness, color, specific identifier, etc., of the border of the remote screen111so that the size of the remote screen111can be easily extracted by an image extraction algorithm.

The mobile terminal120photographs the remote screen111, and displays the captured remote screen111on the display screen121of the mobile terminal120(230). The remote screen111provides content including one or more objects that can be controlled, and the mobile terminal120may display the remote screen111as a camera screen. The operation230may be performed before the operations210and220or simultaneously with the operations210and220.

The display screen121of the mobile terminal120may further include a background screen. The mobile terminal120defines a remote screen area corresponding to the remote screen111, which will be displayed on the display screen121(240). The remote screen area may be defined by a user through an area setting function that is provided by the mobile terminal120. Or, the remote screen area may be defined using a method of extracting a remote screen area by applying an image processing technique such as an edge extraction algorithm to a captured image of the remote screen111, displayed on the display screen121of the mobile terminal120. Or, the mobile terminal120may apply the edge extraction algorithm to a captured image of the remote screen111to define a remote screen area and display the remote screen area on the display screen121, and a user may send a user input to finally decide the remote screen area.

Thereafter, the mobile terminal120performs screen mapping between the remote screen area decided in operation240and the remote screen111, using the screen size of the remote screen111, included in the remote screen resource information (250). The screen mapping means extracting the correspondence relationship between the coordinate values of the remote screen area and the coordinate values of the remote screen111. The mobile terminal120may acquire a conversion matrix related to the rotation angle and scale difference between the remote screen area and remote screen111through the screen mapping. Also, the mobile terminal120may initialize its user input device, for example, a user input interface, such as a touch screen, a mouse, etc.

The mobile terminal120determines whether a user input signal (or, a user input event) is generated through the user input interface, using the camera application (260). If no user input event is generated, the mobile terminal120is maintained in a standby state.

If a user input event is generated, the mobile terminal120checks the generation position of the user input event and the type of the user input event, and generates event information based on the results of the checking (270). In detail, the mobile terminal120converts the generation position of the user input event into the corresponding position on the remote screen111using the conversion matrix acquired in operation250. That is, the mobile terminal120may apply the generation position of the user input event to the conversion matrix to thereby calculate the corresponding coordinate values on the remote screen111. The event information may include information for creating an event for controlling content or an object on the remote screen111. The event information includes the type of the event, an event action corresponding to the event type, and event coordinates that are the corresponding coordinate values on the remote screen111.

The mobile terminal120transmits the event information to the remote apparatus110(280).

Meanwhile, the operation240of defining the remote screen area and the operation250of performing screen mapping between the remote screen area and the remote screen111may be performed whenever it is determined that the mobile terminal120shakes or moves.

If a change in motion of the mobile terminal120is detected, the mobile terminal120may define a new remote screen area, and generate a conversion matrix for converting the coordinates of the newly defined remote screen area into the corresponding coordinates on the remote screen111, thereby updating the conversion matrix according to the detected change in motion of the mobile terminal120. After the conversion matrix is updated, a user input signal that is next received may be processed using the updated conversion matrix.

FIG. 3is a view for explaining the operation240of defining the remote screen area in the display screen121of the mobile terminal120.

If the user executes the remote control mode of the mobile terminal120, the mobile terminal120may photograph a remote screen310of a remote apparatus, and display the captured image of the remote screen310on the display screen121. Since the display screen121includes a background area as well as the remote screen310, the mobile terminal120defines a remote screen area320in the image displayed on the display screen121. As described above, the remote screen area320may be defined automatically using the edge extraction algorithm, or by the user through an input interface, such as a mouse and a touch screen.

FIG. 4is a view for explaining a screen mapping operation corresponding to the operation250ofFIG. 2in detail.

InFIG. 4, a reference numeral400represents the remote screen of a remote apparatus, a reference numeral401represents the display screen of the mobile device120(seeFIG. 1), and a reference numeral402represents a remote screen area that corresponds to the remote screen400projected onto the display screen401of the mobile device120.

In order to perform screen mapping, the sizes and coordinate information of the remote screen400, the display screen401of the mobile terminal120, and the remote screen area402projected onto the display screen401of the mobile terminal120are needed.

In the example ofFIG. 4, it is assumed that the height403and width404of the remote screen400are H and W, respectively, and the height405and width406of the display screen401of the mobile terminal120are h and w, respectively. Also, it is assumed that the coordinates of two vertices of the remote screen area402in a diagonal direction are (x1, y1) and (x2, y2), respectively. Also, the angle of inclination408of the remote screen area402with respect to the display screen401is θ. The angle θ may be, as shown inFIG. 4, defined as the angle of inclination of the remote screen area402with respect to the display screen401in the width direction.

A conversion matrix is created based on the above-mentioned information, and the coordinates (x, y) of the display screen401of the mobile terminal120are applied to the conversion matrix to thereby calculate the coordinates (x′, y′) of the remote screen400. The operation can be expressed as Equation 1, below.
X′=M·X,(1)
where M is the conversion matrix, X is a matrix representing the coordinates (x, y) of the display screen401of the mobile terminal120, and X′ is a matrix representing the coordinates (x′, y′) of the remote screen400.

Equation 1 can be rewritten to Equation 2, below.

FIG. 5is a diagram illustrating an example of the mobile terminal120.

Referring toFIG. 5, the mobile terminal120may include a controller510, a photographing unit520, a display unit530, a communication unit540, a user input unit550, a sensing unit560, and a storage unit570.

Referring toFIGS. 1 and 5, the controller510may be configured to control overall operation of the mobile terminal120. The controller510may control content displayed on a remote screen of the remote apparatus110, and one or more objects configuring the content.

The photographing unit520captures a remote screen that displays content including one or more controllable objects. The photographing unit520may be a camera module.

The display unit530is a display device for displaying an image of the captured remote screen.

The user input unit540receives a user input signal for controlling the objects included in the content. The user input unit540may be a touch screen, a touch pad, a key pad, a mouse, a trackball, etc.

The communication unit540provides a communication interface for communicating with the remote apparatus with the remote screen. The communication unit540may receive remote screen resource information that is to be used to control the objects on the remote screen. The remote screen resource information may include a screen size of the remote screen, and a connection session ID. Also, the communication unit540transmits the results of the controller510processing the user input event to the remote apparatus110.

The controller510defines a remote screen area in the image captured by the photographing unit520. The controller510may define the remote screen area by applying the edge extraction algorithm to the captured image. Or, the controller510may define a remote screen area based on a user input signal. Or, the controller510may apply the edge extraction algorithm to a captured image of the remote screen to define a remote screen area and display the remote screen area through the display unit530, and a user may send a user input to finally decide the remote screen area.

The controller510may perform screen mapping between the remote screen area and the remote screen using the remote screen resource information. The controller510may create a conversion matrix for converting the coordinates of the remote screen area into the corresponding coordinates of the remote screen in order to perform screen mapping between the remote screen area and the remote screen.

The controller510may generate, if it receives a user input signal related to the remote screen area, event information for controlling at least one of the objects, based on the results of the screen mapping. The event information may include an event type, an event action corresponding to the event type, and event coordinates. For this, the controller510may decide the type of an event corresponding to the user input signal, and apply coordinate values corresponding to the user input signal to the conversion matrix to thereby calculate event coordinates that are the corresponding coordinate values of the remote screen.

The controller510may transmit the event information to the remote apparatus110through the communication unit540.

The sensing unit560detects a change in motion of the mobile terminal120, including a change in location or position of the mobile terminal120. In order to detect a change in motion of the mobile terminal120, the sensing unit560may be configured to include a gravity sensor, a gyro sensor, a geo-magnetic sensor, etc. If a change in motion of the mobile terminal120is detected by the sensing unit560, the controller510may define a new remote screen area in the remote screen image displayed on the display unit530, and generate a conversion matrix for converting the coordinates of the newly defined remote screen area into the corresponding coordinates of the remote screen, thereby updating the conversion matrix according to the detected change in motion of the mobile terminal120.

The storage unit570may store an OS of the mobile terminal120, and application programs and data for remote control.

FIG. 6is a flowchart illustrating an example of an operation method of the remote apparatus110with the remote screen.

Referring toFIGS. 1 and 6, the remote apparatus displays content including one or more objects that can be controlled (610). Also, the remote apparatus110may transmit, if it receives a communication connection request from the mobile terminal120, remote screen resource information including a screen size of the remote screen and a connection session ID to the mobile terminal120(610).

Then, the mobile terminal120creates input event information for controlling at least one of the objects, based on the remote screen resource information, and transmits the input event information to the remote apparatus110. That is, the remote apparatus110receives the input event information for controlling at least one of the objects from the mobile terminal120(620).

Then, the remote apparatus110converts the input event information into remote screen event information that can be used in the remote screen (630). The remote apparatus110may search for a conversion adaptor for converting the received input event information into the corresponding remote screen event information in database storing a plurality of conversion adaptors for converting input event information into the corresponding remote screen event information, install the found conversion adaptor, and convert the input event information into the corresponding remote screen event information that can be used in the remote screen, using the installed conversion adaptor.

Next, the remote apparatus110selects an object indicated by the remote screen event information based on representation information about the objects (640).

Then, the remote apparatus110generates a control event for controlling the selected object, and controls the selected object according to the control event (650). Here, the control event is configured to control an object on the remote screen according to remote screen event information and a selection of the object, based on the event information received from the mobile terminal120.

Also, the remote apparatus110may decide, if it receives a plurality of pieces of input event information from a plurality of mobile terminals, a processing order of the plurality of pieces of input event information, convert the plurality of pieces of input event information into the corresponding remote screen event information in the processing order, and control an object(s) based on the remote screen event information.

FIG. 7is a diagram illustrating an example of the remote apparatus110with the remote screen.

Referring toFIG. 7, the remote apparatus110includes a communication connection manager710, a resource manager720, conversion adaptor database730, an input converter740, a screen object information manager750, an input event processor760, a user control manager770, and a screen display unit780.

The communication connection manager710, the resource manager720, the conversion adaptor database730, the input converter740, the screen object information manager750, the input event processor760, and the user control manager770are configured to process input event information received from the mobile terminal120(seeFIG. 1), and to convert the input event information into remote screen event information that can be used by the screen display unit780so that the event according to the input event information is reflected as content or an object(s) displayed on the screen display unit780.

The screen display unit780provides content including one or more objects that can be controlled. The content may be processed by the input event processor760and then provided to the screen display unit780.

The communication connection manager710maintains a communication connection to the mobile terminal120in order to receive a user event from the mobile terminal120through a wired/wireless network. The communication connection manager710may receive input event information for controlling the objects from the mobile terminal120.

The resource manager720may manage resources of the remote apparatus110, the resources including a remote screen size of the screen display unit780. The resource manager720may control, if it receives a communication connection request from the mobile terminal120, the communication connection manager710to provide the mobile terminal120with remote screen resource information that is used to control the objects.

Also, the resource manager720manages conversion adaptors that can be used in the remote screen. The resource manager720may search for, if it receives input event information from the mobile terminal120, an event corresponding to the received input event information from among events that can be used in the remote screen111, and transfer, if the event corresponding to the received input event information is found from among the events that can be used in the remote screen111, the input event information to the input converter740. If the input event information received from the mobile terminal120cannot be processed by the remote apparatus110, the resource manager720may use a conversion adaptor for converting the input event information into the corresponding remote screen event information that can be processed by the remote apparatus110.

The conversion adaptor database730stores at least one conversion adaptor that is used to convert input event information into the corresponding remote screen event information. The conversion adaptor represents information that is used to convert an input event into a remote screen event. The conversion adaptor may be information for converting an event action corresponding to the kind of an input event into an event action corresponding to the kind of the corresponding remote screen event.

The input converter740uses an installed conversion adaptor to convert the input event information into the corresponding remote screen event information that can be used in the remote screen. The resource manager720searches for a conversion adaptor for converting the input event information into the corresponding remote screen event information, in the conversion adaptor database730, and provides the found conversion adaptor to the input converter740. Then, the input converter740may convert the input event information into the corresponding remote screen event information using the conversion adaptor.

The input converter740may be configured with a runtime engine and convert the input event information into the corresponding remote screen event information using the conversion adaptor that is dynamically used. For this operation, the input converter740may install, execute, or terminate at least one conversion adaptor.

The screen object manager750manages representation information of one or more objects, and may select an object indicated by the remote screen event information based on the representation information of the objects. The representation information of each object may include at least one piece of information among the location, size, and height of the object that is displayed on the screen display unit780. The height information of objects may represent a priority order in which the objects appear on a screen.

Since a screen projected onto the mobile terminal120has relatively low resolution, a user may have difficulties in exactly selecting some objects on the screen. That is, a user may fail to select a specific object depending on the representation information of objects. Accordingly, the screen object information manager750may correct coordinate information of the remote screen event information output from the input converter740such that the user can select an exact object depending on the coordinate information of the remote screen event information.

For this, the screen object information manager750may correct coordinate information included in the remote screen event information, based on representation information of the objects included in the content. For example, if event coordinates approximate to representation information of a specific object, the screen object information manager750may correct the event coordinates to coordinates corresponding to the representation information of the specific object. The screen object information manager750may transfer the selected object and the type and coordinate information of the event to the input event processor760.

The input event processor760may generate a control event for controlling the selected object. Also, the input event processor760may generate a control event for controlling an object selected according to the coordinate information corrected by the screen object information manager750. The input event processor760may provide content to the screen display unit780and process the content according to the control event. The control event is a predetermined event for selecting content or at least one object included in the content and changing a screen to a desired form according to the selected content or object. The control event may be generated by calling a predetermined control event based on information received from the screen object information manager750.

If the remote apparatus110is used by a plurality of users, the remote apparatus110may receive two or more input events from two or more users. The user control manager770may maintain and manage user control such that events from the individual users can be sequentially applied to content displayed on the screen display unit780. For this operation, the user control manager770may decide, if it receives a plurality of pieces of input event information from a plurality of mobile terminals, a processing order of the input event information. The input event processor760may control objects according to the processing order decided by the user control manager770.

FIG. 8shows user touch event information received from the mobile terminal120ofFIG. 5.

Referring toFIGS. 1 and 8, when a user input event is generated, the mobile terminal120transmits event information800corresponding to the user input event to the remote apparatus110. The event information800includes the user's identifier (ID)801, an event type802, event action data810, and location information820at which the event action occurred.

The event action data810may include an event's unique name811corresponding to an action, such as touch, click, drag/drop, move, etc., performed by the user, and the type812of the action.

The location information820may include event coordinates that are coordinates to which the event will be applied on a remote screen. There may be provided a plurality of pieces of location information820according to the type of the generated event. For example, in the case of a multi-touch event, since two or more pieces of point location information are generated, the corresponding location information may include all the point location information.

The event information800ofFIG. 8shows a configuration example when the event is a multi-touch event, however, the event information800may be configured in various forms according to the types of events and actions.

FIG. 9shows a configuration example of a conversion adaptor.

The conversion adaptor shown inFIG. 9is used to convert a multi-touch event to a mouse event. In the mobile terminal120(seeFIG. 1), a multi-touch event, such as “touchstart”, “touchmove”, “touchend”, “touchcancel”, etc., may be generated through a multi-touch interface. However, if a desktop computer or the remote apparatus110(seeFIG. 1) with a large screen has no multi-touch interface, the remote apparatus110may convert a multi-touch event received from the mobile terminal120to an event that can be processed by the remote apparatus110using the conversion adaptor ofFIG. 9.

According to the conversion adaptor ofFIG. 9, multi-touch events “touchstart”, “touchmove”, “touchend”, and “touchcancel” are converted into mouse events “mousedown” “mousemove”, “mouseup” and “mouseleave”, respectively.

For example, if a multi-touch event is received from the mobile terminal120and the remote apparatus110supports no multi-touch event, the resource manager720ofFIG. 7may transfer the conversion adaptor as shown inFIG. 9to the input converter740. Then, the input converter740may convert the multi-touch event to a mouse event that can be processed by the remote apparatus110, using the conversion adaptor. The screen object information manager750corrects the coordinates of the mouse event, and transfers information about an object indicated by the corrected coordinates, the mouse event, the corrected coordinate information, etc. to the input event processor760. Then, the input event processor760finally generates a mouse event based on the received information, and may control an object related to the mouse event.

Therefore, according to the examples described above, a user can easily select and control applications or content displayed on a remote screen, using a camera, while viewing the remote screen. Also, by supporting real-time conversion between input events through interface conversion it is possible to remotely control various kinds of devices using a smart phone that a user can always carry.

The present invention can be implemented as computer-readable code in a computer-readable recording medium. The computer-readable recording medium includes all types of recording media in which computer-readable data are stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage. Further, the recording medium may be implemented in the form of carrier waves such as used in Internet transmission. In addition, the computer-readable recording medium may be distributed to computer systems over a network, in which computer-readable code may be stored and executed in a distributed manner.