Method and apparatus for displaying a pointer on an external display

The present invention relates to a method of displaying a pointer on an external device sharing an output video with a terminal, and an apparatus thereof. A method of controlling display of a pointer by a terminal so that the pointer is displayed on an external device according to an embodiment of the present invention includes transmitting an output video of the terminal to the external device, if a predetermined request exists; detecting a sensing value according to a input performed on a display area; generating a pointer displaying signal used for displaying a pointer at a position corresponding to a position where the detected sensing value is detected on the external device based on the detected sensing value; and transmitting the generated pointer displaying signal to the external device. According to the present invention, since a pointer is displayed on an external device sharing an output video with a terminal, the user may execute a desired function by looking at the external device, without looking at the terminal.

CROSS RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. §119(a) to Korean Application Serial No. 10-2013-0011356, which was filed in the Korean Intellectual Property Office on Jan. 31, 2013, the entire content of which is hereby incorporated by reference.

BACKGROUND

The present disclosure relates to a method of displaying a pointer on an external device sharing an output video with a terminal, and an apparatus thereof.

2. Description of the Related Art

According to the recent development of multimedia technologies, various terminals such as smart phones are widely distributed, and such terminals are equipped with various applications that output videos.

Further, the development of the communication technologies enables the terminals to communicate with various external devices such as monitors, and accordingly the terminals and monitors are capable of sharing output videos.

In order to share the output videos like this, communication technologies such as MHL (Mobile High-definition Link) and Wi-Fi P2P (Peer to Peer) have been used. More particularly, a terminal and the external device may share the output videos using various communication technologies as described above. The sharing of the output video provides convenience for a user for watching a video, by enabling the video output in the terminal in a relatively small size to be output to a larger screen, e.g., monitor.

Meanwhile, while the output video is shared, if a user executes an application existing at a specific position or selects a button to which a specific function is assigned, the user may perform an operation while watching the video output on the terminal. This is described with reference toFIG. 1.

For example, as illustrated inFIG. 1, if a terminal100and an external device200are connected by MHL, the external device200may output or otherwise display a video the same as the video being output by the terminal100.

At this point, if a user desires to select a specific application while watching a video being output in the external device200, since a position of an electronic pen110or a finger of the user (not illustrated) is not displayed on the external device200, the user may not select the application by looking at the external device200.

Therefore, the user must select the application to be executed by looking at the terminal100, and this inconveniences the user to look at both of the terminal100and the external device200, alternately.

SUMMARY

Accordingly, the present disclosure provides a method that enables a pointer to be displayed on an external device, when an output video is shared between a terminal and the external device.

In order to accomplish this object, an embodiment of the present disclosure provides a method of controlling display of a pointer by a terminal so that the pointer is displayed on an external device. The method includes transmitting an output video of the terminal to the external device; detecting a sensing value with an input unit according to a input performed on a display area; generating a pointer displaying signal used for displaying a pointer on the external device at a position corresponding to a position where the detected sensing value is detected based on the detected sensing value; and transmitting the generated pointer displaying signal to the external device.

Meanwhile, another aspect provides a method of displaying a pointer on an external device while outputting a video received from a terminal. The method includes outputting a video received from the terminal on a display area; determining a position of the pointer to be displayed on the display area based on a pointer displaying signal received from the external device during output of the video; and displaying the pointer on the determined position.

Meanwhile, another aspect provides an apparatus that controls display of a pointer by a terminal so that the pointer is displayed on an external device. The apparatus includes a communicating unit that performs a communication with the external device; an input and output unit that outputs a video and detect a sensing value according to a input; and a controller that transmits an output video of the terminal which is being output through the input and output unit to the external device through the communicating unit, generates a pointer displaying signal to be used for displaying the pointer at a position corresponding to a position on the external device where the sensing value is detected based on the sensing value received from the input and output unit, and transmits the generated pointer displaying signal through the communicating unit to the external device.

Meanwhile, another aspect provides an apparatus that displays a pointer on an external device currently outputting a video received from a terminal. The apparatus includes a communicating unit that performs a communication with a terminal; an output unit that outputs a video; and a controller that outputs a video received from the terminal via the communicating unit through the output unit, determines a position of the pointer to be displayed on the output unit based on a pointer displaying signal received from the external device via the communicating unit during output of the video, and then displays the pointer on the determined position.

The pointer is preferably displayed on an external device that shares an output video with a terminal, so the user may execute a desired function by looking at only the external device without looking at the terminal.

DETAILED DESCRIPTION

Hereinafter, in the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

As described above, while the terminal and the external device output the same video, a user who watches the video via the external device does not know at which point of the output video being output a finger or an input unit such as an electronic pen is positioned, and therefore the user may not be able to execute a specific application or select a menu having a specific function, via the external device.

Therefore, in preferred embodiments the present invention provides a method that enables a user to execute a specific application or select a menu having a specific function, only through the external device, by displaying a pointer indicating a present position of the input unit on the external device.

One embodiment of a method of displaying a location of a pointer according to the embodiment of the present invention will be described with reference toFIG. 2.

In step201, when a terminal100receives a video transmission request, the terminal100proceeds to step203to transmit a requested video to an external device200. Here, the video transmission request preferably refers to a request to transmit a video which is being output by the terminal100, to the external device200.

In step205, the external device200outputs a video received from the terminal100. Typically, the external device200continuously receives the video from the terminal100and is being output the video, rather than having the video information being transmitted by the terminal100and cached in memory of the external device200for display.

In step207, the terminal100detect a sensing value according to input (e.g., user input) on a display area. The sensing value may be a sensing value with respect to any one of capacitance, magnetic field, and pressure, and preferably the sensing value corresponds to a particular location of pointer or selection device on the display area.

In step209, the terminal100generates a pointer displaying signal to be used for indicating a pointer on the external device200based on the detected sensing value, and then the process proceeds to step211. The pointer displaying signal is typically a signal to indicate the pointer on a display area of the external device200corresponding to a position where the sensing value is detected, and includes coordinate information at a position where the sensing value has been sensed. Even though the invention is generally described with the terminal100detecting contact between the pointer and the terminal100, it is understood that such contact can actually be without direct contact, e.g., the two devices being within a small distance, e.g., 5-10 mm.

In step211, the terminal100transmits the generated pointer displaying signal to the external device200.

In step213, the external device200analyzes the pointer displaying signal received from the terminal100and displays the pointer on the display area based on the analyzed result. That is, the external device200indicates the pointer at a position corresponding to the user input of the terminal100.

According to the embodiment as described above, since the position of the input unit, that is, the position of a finger of the user or an electronic pen, is displayed on the external device200, the user may execute a specific application or select a menu having a specific function without looking at the terminal100.

In the above, the concept of the method of displaying the point according to the embodiment of the present invention has been described with reference toFIG. 2. Hereinafter, embodiments of the present invention will be described in detail with reference to corresponding drawings.

FIG. 3is a flowchart illustrating a method of controlling display of a pointer in a terminal according to embodiments of the present invention.

In step301, the terminal determines whether the video transmission request to an external device exists, and if the video transmission request to the external device exists, a terminal proceeds to step303. Here, the video transmission request refers to a request to transmit a video which is being output by the terminal to the external device. The request may be automatically performed when a session with the external device has been established, for example, by using various communication technologies. For example, when an MHL transmission cable has been connected between the terminal and the external device to establish a session between the terminal and the external device, or when a Wi-Fi P2P session between the terminal and the external device has been established, it may be assumed that such request has been made. Otherwise, when the user selects a specific menu after the establishment of the session with the external device, it may be assumed that such request has been made.

Although the invention is described as being used in combination with an MHL transmission cable and Wi-Fi P2P, the invention is not limited to such transmission technologies or protocols. For example, any other type of transmission system is acceptable, including, but not limited to the presently existing Bluetooth, Ethernet, 802.11a/b/g/n/ac, as well as cellular communication technologies or any future developed wired or wireless communication system capable of handling the video as well as the pointer displaying signal.

In step303, after the terminal transmits the video that the terminal is currently being output, to the external device, the terminal proceeds to step305. In the following steps, it is assumed that the video that is currently being output from the terminal is continuously transmitted to the external device. Meanwhile, in step303, the terminal may change its own state of display. For example, the terminal may reduce power consumption by reducing the brightness of the output video or by cutting off power supply for displaying the output video.

In step307, the terminal detects a sensing value with respect to the user input from the display area. The sensing value may be a sensing value with respect to any one of capacitance, magnetic field, and pressure.

In step307, the terminal generates a pointer displaying signal based on the detected sensing value, and transmits the generated pointer displaying signal to the external device. In generating the pointer displaying signal, the terminal compares the detected sensing value with a predetermined threshold value and generates the pointer displaying signal if the sensing value is equal to or greater than the predetermined threshold value. By using the predetermined threshold value, many inadvertent or unintentional sensed sensing values can be ignored.

In the above, the operation in the terminal according to an embodiment of the present invention has been described with reference toFIG. 3. Hereinafter, an operation in the external device according to an embodiment of the present invention will be described in detail with reference to corresponding drawings.

FIG. 4is a flowchart illustrating a method of displaying a pointer in an external device according to an embodiment of the present invention.

In step401, the external device outputs or displays a video received from a terminal. At this point, the video received from the terminal is a video which is currently being output by the terminal, and, in the following steps, it is assumed that the external device continuously receives and outputs the video currently being output on the terminal. Meanwhile, the external device may adjust the size of the video in accordance with its own resolution in outputting the video received from the terminal.

In step403, the external device receives the pointer displaying signal from the terminal, and the external device proceeds to step405.

In step405, the external device analyzes the pointer displaying signal received from the terminal, and displays the pointer based on the analysis result. That is, the external device determines at which point of the display area the pointer is to be displayed and displays the pointer based on the determination.

In the above, a method of displaying the pointer on the external device according to an embodiment of the present invention has been described with reference toFIGS. 3 and 4. Examples according to the method will be described with reference toFIGS. 5A and 5B.

FIG. 5Ais a diagram illustrating a case in which a user executes a specific application App3 using an electronic pen110, while a video is being output on both of a terminal100and an external device200. When the user positions the electronic pen110at a specific position of the terminal100, a pointer202indicating the position of the electronic pen110is displayed on the external device200. Therefore, the user may know the position of the electronic pen110without looking at the terminal100. In addition, if the pointer is displayed on an icon corresponding to the application App3 that the user desires to execute, the user may select and execute the corresponding application App3.

FIG. 5Bis a diagram illustrating a case in which the user selects a specific menu by using the electronic pen110while the video is being output on both of the terminal100and the external device200. As described with reference toFIG. 5A, since the pointer202indicating the current position of the electronic pen110is displayed on the external device200, the user may know the current position of the electronic pen110only by looking at the external device200, and accordingly the user may execute the corresponding function by selecting the menu to perform the specific function.

In the above, the method of displaying the pointer on the external device200according to an embodiment of the present invention has been described with reference toFIGS. 3 to 5. Hereinafter, a method of displaying a pointer on the external device200according to another embodiment of the present invention will be described with reference to the corresponding drawings.

FIG. 6is a flowchart illustrating a method of controlling display of a pointer according to another embodiment of the present invention.

Since steps601to605are the same as steps401to405described with reference toFIG. 4, the detailed description thereof will not be provided here.

In step607, a terminal determines whether a sensing value detected by user input is equal to or more than a predetermined threshold value. The terminal proceeds to step609only if the detected sensing value is equal to or more than the predetermined threshold value, and otherwise, the terminal proceeds to step605.

In step609, the terminal generates a pointer displaying signal, transmits the generated pointer displaying signal to an external device, and proceeds to step611.

In step611, the terminal determines whether the sensing value equal to or more than the threshold value is detected for the predetermined time or more at the same position. The terminal proceeds to step613only if the sensing value is equal to or more than the threshold value is detected for the predetermined time or more at the same position, or otherwise the returns to step605.

In step613, the terminal performs a function corresponding to the position where the detected sensing value is detected. For example, if an icon of a specific application is displayed at a position where the sensing value is detected, the corresponding application is performed, or if a menu for performing a specific function is displayed at a position where the sensing value is detected, the corresponding function is performed.

In the above, the method of generating the pointer displaying signal or performing the specific function by using one threshold value has been described with reference toFIG. 6. Hereinafter, a method of generating the pointer displaying signal or performing the specific function by using two threshold values will be described.

FIG. 7is a flowchart illustrating a method of controlling display of a pointer according to another embodiment of the present invention.

Since steps701to705are the same as steps401to405as described with reference toFIG. 4, the detailed description thereof will not be provided here.

In step707, the terminal compares an detected sensing value S and predetermined threshold values Th1and Th2.

If it is determined that the detected sensing value is S less than the first threshold value Th1based on the determination result in step707, the terminal performs no function and proceeds to step705.

Meanwhile, if it is determined that the detected sensing value is equal to or more than the first threshold value Th1and less than the second threshold value Th2based on the determination result in step707, the terminal proceeds to step711in which the terminal generates a pointer displaying signal, transmits the generated pointer displaying signal to the external device, and then proceeds to step705.

Meanwhile, if it is determined that the detected sensing value is equal to or more than the second threshold value Th2based on the determination result in step707, the terminal proceeds to step721, in which the terminal performs a function corresponding to the position where the sensing value is detected, and then proceeds to step705. That is, in step721, the terminal executes a specific application or performs a function corresponding to the corresponding position, and then proceeds to step705. More particularly, in this embodiment, if the sensing value is less than Th1, the input is ignored, if the sensing value is between Th1and Th2, the pointer is displayed, but no action is taken. Only if the sensing value is greater than Th1is the function performed.

The embodiment with reference toFIG. 7has been described under the assumption that a pointer in the same type is displayed regardless of a duration of user input. Pointers may be displayed in different types according to the duration of user input depending on embodiments. This will be described with reference toFIG. 8.

FIG. 8is a flowchart illustrating a method of controlling display of a pointer according to another embodiment of the present invention.

Since steps801to805ofFIG. 8are the same as steps401to405described with reference toFIG. 4, the detailed description thereof will not be provided.

In step807, the terminal compares the detected sensing value with the predetermined threshold values Th1and Th2.

If it is determined that the detected sensing value is equal to or more than the predetermined first threshold value Th1, and lower than the predetermined second threshold value Th2based on the determination result in step807, the terminal proceeds to step811, in which the terminal generates a pointer displaying signal to display a first type of pointer, transmits the generated pointer displaying signal to the external device, and then proceeds to step805.

If it is determined that the detected sensing value is equal to or more than the predetermined second threshold value Th2based on the determination result in step807, the terminal proceeds to step821, in which the terminal determines whether the detected sensing value is continuously detected at the same position for longer than a predetermined time.

If it is determined that the sensing value which is equal to or more than the second threshold value Th2is detected at the same position for equal to or longer than the predetermined time based on the determination result in step821, the terminal proceeds to step823, in which the terminal generates a pointer displaying signal to display a second type of pointer, transmits the generated pointer displaying signal to the external device, and then proceeds to step805.

Here, the second type of pointer refers to a pointer which is different from the first type of pointer in at least one of shape, color, opacity or size. Therefore, according to the present embodiment of the present invention, the pointer displaying signal includes information with respect to a position where the pointer is to be displayed and information with respect to a type of pointer.

Meanwhile, if it is determined that the sensing value which is equal to or more than the second threshold value Th2is detected at the same position for shorter than the predetermined time based on the determination result in step821, the terminal proceeds to step825, in which the terminal performs a function corresponding to the corresponding position, and proceeds to step805. More particularly, the function is performed only when the sensing value is detected for a time less than the predetermined time. This embodiment allows for situations when the first type of pointer, as initially displayed on the external device, is not visualized by the user (e.g., obscured by the background image or the displayed pointer does not have enough contrast) or when the sensing value, although correctly measured by the terminal, was inadvertent.

According to the embodiment, the terminal may perform the function corresponding to the corresponding position in step823, and generate the pointer displaying signal to display the second type of pointer in step825.

In the above, the method of controlling the display of a pointer and the method of displaying the pointer have been described with reference toFIGS. 2 to 8. Hereinafter, structures of the terminal and the external device to which the embodiments of the present invention are applied will be described with reference to the corresponding drawings.

FIG. 9is a block diagram illustrating a typical terminal to which embodiments of the present invention may be applied.

With reference toFIG. 9, the terminal to which the embodiments of the present invention are applied typically includes a controller910, a communicating unit920, and an input and output unit930. The terminal100may further include a memory unit940, depending on the embodiment.

If a predetermined request is made, the controller910transmits its own output video which is being output through an input and output unit930to the external device via the communicating unit920. If a selection of a predetermined menu by a user is made or a session with the external device is established, the controller910may determine that the predetermined request has been made. Meanwhile, the transmission of the output video may be continued until a specific interruption, such as a selection of a predetermined menu or release of the session, is generated.

The controller910generates the pointer displaying signal used for displaying a pointer on the external device based on a sensing value received from the input and output unit930, and transmits the generated pointer displaying signal through the communicating unit920to the external device. The sensing value may be at least one of capacitance, a magnetic field and pressure.

In an embodiment, the controller910compares the sensing value received from the input and output unit930with the predetermined threshold value, and generates a pointer displaying signal if the sensing value is more than the predetermined threshold value. At this point, the controller910may perform the function corresponding to the corresponding position when the sensing value which is equal to or more than the predetermined threshold value is detected at the same position for equal to or more than a predetermined time. For example, when an icon indicating a specific application exists at the corresponding position, the controller910may execute the corresponding application. Otherwise, when a menu to perform a specific function may be displayed at the corresponding position, the controller910may perform a function corresponding to the corresponding menu.

The embodiments will be described with reference toFIGS. 10A to 10Das follows.

FIGS. 10A and 10Bare diagrams illustrating embodiments of using a change of capacitance values, andFIGS. 10C and 10Dare diagrams illustrating embodiments of using pen pressure. As illustrated inFIGS. 10A and 10C, if a sensing value detected in the input and output unit930, that is, a change amount of the capacitance or the pen pressure, is smaller than a predetermined threshold value Th, the terminal does not generate a pointer displaying signal, so no pointer is displayed on the external device200. In addition, as illustrated inFIGS. 10B and 10D, if a sensing value detected in the input and output unit930, that is, a change amount of the capacitance or the pen pressure, is larger than the predetermined threshold value Th, the terminal generates a pointer displaying signal and transmits the generated pointer displaying signal so that a pointer202is displayed on the external device200. If a sensing value equal to or more than the threshold value Th is detected at the same position for equal to or longer than the predetermined time, the terminal performs a function corresponding to the corresponding position.

In another embodiment, the controller910may compare a sensing value received from the input and output unit930with the predetermined first and second threshold values, and generate a pointer displaying signal to display a first type of pointer if the sensing value is equal to or more than the predetermined first threshold value and less than the predetermined second threshold value. In addition, the controller910may generate a pointer displaying signal to display a second type of pointer or a function corresponding to the corresponding position if the sensing value is equal to or more than the predetermined second threshold value. Further, if the sensing value is equal to or more than the predetermined second threshold value, the controller910may determine whether the sensing value is detected in the same position for equal to or longer than a predetermined time, and generate a pointer displaying signal to display a second type of pointer if the sensing value is detected in the same position for equal to or longer than the predetermined time. Here, the second type of pointer may be a pointer which is different from the first type of pointer in at least one of shape, color, or opacity.

The embodiments are described with reference toFIGS. 11A to 11Das follows.

FIGS. 11A to 11Care diagrams illustrating an embodiment of using a change of capacitance values, andFIGS. 11D to 11Fare diagrams illustrating an embodiment of using pen pressure. As illustrated inFIGS. 11A and 11C, if a sensing value detected in the input and output unit930, that is, a change amount of the capacitance or the pen pressure, is smaller than a predetermined first threshold value Th1, the terminal does not generate a pointer displaying signal, so no pointer is displayed on the external device200. In addition, as illustrated inFIGS. 11B and 11D, if a sensing value detected in the input and output unit930, that is, a change amount of the capacitance or the pen pressure, is equal to or larger than the predetermined first threshold value Th1and smaller than a predetermined second threshold value Th2, the terminal generates a pointer displaying signal to display a first type of pointer and transmits the generated pointer displaying signal to the external device, so the first type of pointer202is displayed on the external device200. As illustrated inFIGS. 11C and 11D, if a sensing value detected in the input and output unit930, that is, a change amount of the capacitance or the pen pressure, is equal to or larger than the predetermined second threshold value Th2, the terminal generates a pointer displaying signal to display a second type of pointer and transmits the generated pointer displaying signal to the external device, so the second type of pointer202is displayed on the external device200. In addition, if a sensing value equal to or more than the second threshold value Th2is detected on the same position for equal to or longer than a predetermined time, the terminal performs a function corresponding to the corresponding position. According to the embodiment, if a sensing value equal to or more than the second threshold value Th2is detected on the same position for equal to or longer than the predetermined time, the terminal performs a function corresponding to the corresponding position, and if a sensing value equal to or more than the second threshold value Th2is detected on the same position for shorter than the predetermined time, the terminal may generate and transmit a pointer displaying signal to display the second type of pointer.

Meanwhile, the controller910may change the display state of the input and output unit930while the output video is transmitted to the external device200. For example, the controller910may lower the brightness of the output video, or cut off power supply for display of the output video.

The communicating unit920performs a communication with the external device by various communication methods, for example, MHL or Wi-Fi P2P, under the control of the controller910. Further, the communicating unit920transmits the video currently being output and the pointer displaying signal to the external device under the control of the controller910.

The input and output unit930may include a displaying unit that displays an output video and a separate sensing unit that senses input by a user. The sensing unit may use a unit that senses at least one of capacitance, magnetic field, and pressure.

The memory unit940may be used to store, e.g., the threshold values to be compared with the sensing value.

In the above, a description is made of the terminal to which the embodiments of the present invention is applied, with reference toFIG. 9. Hereinafter, a description will be made of the external device to which the embodiments of the present invention are applied with reference toFIG. 12.

FIG. 12is a block diagram illustrating an external device to which a subset of embodiments of the present invention may apply.

With reference toFIG. 12, the external device to which the embodiments of the present invention are applied includes a controller1210, a communicating unit1220, and an output unit1230.

The controller1210outputs a video through the output unit1230, which is received from the terminal via the communicating unit1220, determines the position of a pointer displayed on the output unit1230based on the pointer displaying signal received from the terminal through the communicating unit1220while the video is being output, and then displays the pointer on the determined position. At this point, the controller1210may further determine the type of pointer to be displayed on the output unit based on the pointer displaying signal.

The communicating unit1220performs a communication with the terminal by various communication methods, for example, MHL or Wi-Fi P2P, under the control of the controller1210. Further, the communicating unit1220displays the pointer on the video currently being output under the control of the controller1210.

Under the control of the controller1210, the output unit1230outputs the video received from the terminal and displays the pointer.

Embodiments of the present invention described above may be implemented any of various methods. For example, embodiments of the present invention may be implemented by combinations of hardware and, software. When the embodiments of the present invention are implemented utilizing software, the software may be executed on one or more processors using various operating systems and platforms. In addition, such software may be prepared using any programming languages out of a plurality of appropriate programming languages, and may be compiled to executable machine language codes or intermediate codes that are executed on a framework or a virtual machine.

The above-described methods according to the present invention can be implemented in hardware, firmware or via the execution of software or computer code that is stored on a non-transitory machine readable medium such as a CD ROM, a RAM, a floppy disk, a hard disk, or a magneto-optical disk or computer code downloaded over a network originally stored on a remote recording medium or a non-transitory machine readable medium and stored on a local non-transitory recording medium, so that the methods described herein are loaded into hardware such as a general purpose computer, or a special processor or in programmable or dedicated hardware, such as an ASIC or FPGA. As would be understood in the art, the computer, the processor, microprocessor controller or the programmable hardware include memory components, e.g., RAM, ROM, Flash, etc. that may store or receive software or computer code that when accessed and executed by the computer, processor or hardware implement the processing methods described herein. In addition, it would be recognized that when a general purpose computer accesses code for implementing the processing shown herein, the execution of the code transforms the general purpose computer into a special purpose computer for executing the processing shown herein. In addition, an artisan understands and appreciates that a “processor” or “microprocessor” constitute hardware in the claimed invention. Under the broadest reasonable interpretation, the appended claims constitute statutory subject matter in compliance with 35 U.S.C. §101 and none of the elements consist of software per se.

Further, when the embodiments of the present invention are executed on one or more processors, the embodiments may be implemented by a processor-readable medium (for example, a memory, a floppy disc, a hard disc, a compact disc, an optical disc, or a magnetic tape) which stores one or more programs that perform methods of implementing various embodiments of the present invention discussed above.