Patent Description:
Smart TVs have replaced traditional TVs and can be equipped with a wide range of programmes and applications for users to choose from and watch. Smart TVs are controlled by a remote control, which usually has only four directional keys (up, down, left and right) to control the direction, making interaction inefficient and time-consuming.

<CIT> describes a system and method for executing a command according to a worker's gesture by recognizing the worker's gesture in an environment where the worker cannot directly touch and control equipment. <CIT> describes a method, comprising tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display (NED), wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly; observing a gesture of one hand of the user by a second camera, and executing a further operation if at least one gesture of the hand is detected. <CIT> describes a technology for calculating spatial trajectories of different gestures and determining a dominant gesture based on magnitudes of the spatial trajectories. <CIT> describes an information processing apparatus includes a memory storing a program and at least one processor that executes the program to implement processes of detecting a speed of motion of a user based on motion information relating to a motion of the user that is detected by a detection device, and generating position information of a position indication display information item, which is displayed on a display device and indicates a position designated by the user, based on the motion information relating to the motion of the user.

This summary part is provided to introduce concepts in a brief form, and these concepts will be further described in the following specific embodiments. The summary is intended to neither identify key features or essential features of the claimed technical solutions nor limit the scope of the claimed technical solutions.

The above and other features, advantages and aspects of embodiments of the disclosure will become more apparent in combination with the accompanying drawings and with reference to the following specific implementations. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the accompanying drawings are illustrative, and the originals and elements are not necessarily drawn to scale.

The embodiments of the disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the disclosure are shown in the accompanying drawings, it should be understood that the disclosure may be implemented in various forms and should not be construed as being limited to the embodiments described herein, on the contrary, these embodiments are provided for a more thorough and complete understanding of the disclosure.

It should be understood that the steps described in the embodiments of the disclosure may be performed according to different orders and/or in parallel. In addition, the embodiments may include additional steps and/or omit the execution of the shown steps.

In the following description, although numerous features may be designated as optional, it is nevertheless acknowledged that all features defined by the independent claims are not to be read as optional.

The term "comprising" used herein and variants thereof means open-ended including, i.e., "including, but not limited to". The term "based on" refers to "based at least in part on". The term "one embodiment" represents "at least one embodiment"; the term "the other embodiment" represents "at least one additional embodiment"; and the term "some embodiments" represents "at least some embodiments". Definitions of other terms will be provided in the description below.

It should be noted that the terms such as "first", "second" and the like mentioned in the disclosure are merely intended to distinguish different devices, modules or units, rather than limiting an order of functions executed by these devices, modules or units or an interdependence among these devices, modules or units.

It should be noted that the modifications of "a" and "multiple" mentioned in the disclosure are illustrative, but are not restrictive. It should be understood by those skilled in the art that it should be understood as "one or more" unless otherwise specified in the context.

Names of messages or information interacted among a plurality of devices in the embodiments of the disclosure are merely for an illustrative purpose, rather than limiting the scope of these messages or information.

Referring to <FIG>, which illustrates a flow chart of a control method <NUM> provided according to an embodiment of the disclosure, the method <NUM> may be used for a terminal device including but not limited to a far-field display device. The far-field display device refers to a display device that cannot be controlled by the user in a direct contact manner using body parts or other physical control devices such as a stylus, including but not limited to electronic devices such as TVs and conference screens. Specifically, the method <NUM> includes step S101 to step S104:
step S101, an image captured by a camera is received.

The camera may be built in or externally connected to the terminal device, which may send captured image data to the terminal device in real time for processing. Advantageously, the camera is set in a way that may face the user directly, so as to capture limb instructions sent by the user to the terminal device.

It needs to be noted that in other embodiments, images may further be received in other manners, or images captured or transmitted by other apparatuses are received, which is not limited in the disclosure.

Step S102, position information of a first part and gesture information of a second part of a user are obtained based on the image.

The first part and the second part are body parts of the user, such as one or more hands. The position information of the first part is used to describe the position of the first part in the image, or the position of the first part relative to the controlled terminal device, and gesture information of the second part is used to describe the gesture of the second part, such as a hand gesture, etc..

Exemplarily, the position information of the first part and the gesture information of the second part of the user in the image may be obtained.

Step S103, a movement trajectory of a navigation indicator is determined based on the position information of the first part.

The navigation indicator may be used for selecting and controlling a visual element on a display interface. The navigation indicator may be represented by an icon, such as a cursor, or pointer, for example. and the navigation indicator may also be hidden while the visual element is highlighted or otherwise animated to indicate that the visual element is selected. The movement trajectory of the navigation indicator includes one or a group of moving vectors, which reflect a moving displacement and direction of the navigation indicator. The movement trajectory of the navigation indicator is determined by the position information of the first part of the user.

Exemplarily, a controlled element to which the navigation indicator is directed may be determined based on the position information of the first part, for example, a position and/or a movement trajectory of the navigation indicator on a controlled device is determined based on the position information of the first part relative to the controlled device, and the controlled element to which the navigation indicator is directed is determined based on the position and/or the movement trajectory.

Step S104, a control command is determined based on the gesture information of the second part, and the control command is used for controlling a visual element to which the navigation indicator is directed.

The control command is used to control or perform operation on the visual element to which the navigation indicator is directed, including clicking, touching, long pressing, zooming in, zooming out and rotating the visual element. In some embodiments, a mapping relation between the gesture information of each second part and the control command may be preset, so that the control command corresponding to the gesture information of the second part may be determined based on the mapping relation.

In this way, according to the control method provided by the embodiment of the disclosure, the movement trajectory of the navigation indicator is determined based on the position information of the first part, and the control command is determined based on the gesture information of the second part, so that determination of the control command is independent of determination of the position of the navigation indicator. On the one hand, the determination of the control command is based on static gesture information, while the determination of the position of the navigation indicator is based on dynamic position changes, thus facilitating the use of different characteristic algorithms to determine the above two processes respectively. Exemplarily, the determination of the control command may be based on the static gesture information, while the determination of the position of the navigation indicator is based on the dynamic position change information. Therefore, for the above two different calculation characteristics, the position information of the first part and the gesture information of the second part may be calculated by adopting computing modules with corresponding characteristics respectively, thereby improving the targeting of the information acquisition, the accuracy of the calculation and the utilization of the calculation resources. On the other hand, the determination of the control command and the determination of the position of the navigation indicator are based on different body parts of the user, which can make the determination processes of the two not affect each other, especially the shape of the contour of the first part does not change with the gesture of the second part, which can avoid the change of the gesture affecting the movement of the navigation indicator, and thus can improve the recognition accuracy of the user command.

According to the invention, the first part and the second part belong to different body parts of the same user. There is no inclusive relationship between the first part and the second part, for example, when the second part is the hand, the first part can be the wrist, the elbow and not the fingers. In the embodiment of the disclosure, the movement trajectory of the navigation indicator and the control command are determined based on different body parts of the user respectively, so that the determination of the control command can be prevented from being affected when the user changes the position of the first part or the determination of the movement trajectory of the navigation indicator can be prevented from being affected when the user changes the gesture of the second part.

In some embodiments, the position of the second part can change with the position of the first part; and a position or gesture of the first part itself does not affect the gesture of the second part. In this way, the position of the second part can follow that of the first part, allowing the first and second parts to move in an interconnected space, avoiding the difficulty of capturing images of both the first and second parts due to the large spatial distance between them, which makes it difficult for the camera device to capture images of the first and second parts at the same time, thus increasing the success rate and ease of control of the controlled element using the first and second parts. In addition, changes in the position and/or gesture the first part do not affect the gesture of the second part, which also improves the accuracy of the control commands generated based on the attitude of the second part, allowing precise and easy control of the position of the navigation indicator and the issuing of control commands.

According to the invention, the first part comprises a wrist, and the second part comprises a hand. In the embodiment of the disclosure, the wrist reflects the movement of the gesture accurately and steadily, and is less affected by changes in the gesture than the fingers or palm of the hand, allowing precise control of the movement of the navigation indicator. The movement of the wrist has no effect on the gesture, so that control commands can be given easily and precisely.

In some embodiments, step S <NUM> further includes:.

The determination of the control command is based on the static gesture information, while the determination of the position of the navigation indicator is based on the dynamic position change. Therefore, in this embodiment, the position information of the first part and the gesture information of the second part are calculated by adopting the computing modules with different characteristics respectively, the pertinence of information acquisition can be improved, thus increasing the calculation accuracy and the utilization of calculation resources.

In some embodiments, the first computing module may run a first machine learning model, and the second computing module may run a second machine learning model. The first machine learning module and the second machine learning module are trained to reliably recognize and distinguish the first part and the second part of the user. By using a trained machine learning model to determine the position information of the first part and the gesture information of the second part, recognition accuracy can be improved and computational resources and hardware costs can be reduced.

In some embodiments, step S104 further includes:
step B <NUM>, if the gesture information of the second part conforms to a preset first gesture, a controlled element is controlled based on the gesture information of the second part.

The first gesture may include one or more preset hand gestures.

In some embodiments, step S104 further includes:
step B2, if the gesture information of the second part does not conform to the preset first gesture, the controlled element is not controlled based on the gesture information of the second part.

In some embodiments, when the gesture information of the second part does not conform to the preset first gesture, the navigation indicator is moved only based on the position information of the first part.

In some embodiments, the method <NUM> further includes:
step S105, the visual element to which the navigation indicator is directed is controlled based on position information of the first part obtained based on at least two frames of target image. Exemplarily, the controlled element to which the navigation indicator is directed may be controlled based on the position change information of the first part obtained from the at least two frames of target image. The manner to control the controlled element to which the navigation indicator is directed includes, but not limited to controlling the controlled element to move on a controlled device by scrolling or moving, for example, scrolling or moving an application interface, an icon or other controls.

A method of determining the at least two frames of target image includes:.

According to one or more embodiments of the disclosure, the target image is an image whose gesture information conforms to the second gesture. According to the invention, by translating the change in position of the first part into a scrolling effect of the visual element when the gesture information conforms to the second gesture, the user can control the navigation indicator to scroll the visual element, thereby enhancing interaction efficiency. The second gesture may include one or more preset hand gestures. Exemplarily, the controlled element may be moved based on the position information of the first part and/or the preset gesture of the second part, so that the controlled element to which the navigation indicator is directed is determined.

In some embodiments, step S105 further includes:.

The motion information of the first part includes one or more of the following: motion time of the first part, a motion speed of the first part, a displacement of the first part and a motion acceleration of the first part. In this embodiment, the motion information is determined based on the position information, and initial parameters and conditions required for scrolling the visual element can be obtained, so that relevant scrolling parameters of the visual element are determined.

In some embodiments, step E2 further includes:.

In some embodiments, the second gesture is that a preset number of fingers splay out. Exemplarily, the second gesture comprises splaying five fingers of one hand apart. The scrolling commands usually require fast movement of the gesture, and with fast movement, splaying a preset number of fingers apart is easier to recognize than other gestures, thus improving recognition accuracy.

In some embodiments, step S103 further includes: if the gesture information of the second part conforms to a preset third gesture, the movement trajectory of the navigation indicator is determined based on the position information of the first part. The third gesture may include a plurality of preset hand gestures. In this embodiment, if the gesture information of the second part conforms to the preset third gesture, the movement trajectory of the navigation indicator is determined based on the position information of first part. For example, the navigation indicator is moved only based on the position of the first part when the hand conforming to the preset hand gesture, which prevent the user from unintentionally moving the first part to cause the navigation indicator to move by mistake.

In some embodiment, step S103 further includes: the movement trajectory of the navigation indicator is determined based on the position information of the first part obtained from spaced images. In the embodiment of the disclosure, in order to prevent jitter of the navigation indicator caused by the inevitable shaking of the user when waving the first part, the movement trajectory of the navigation indicator can be determined based on the position information of the first part obtained from the spaced images, which can reduce jitter of the navigation indicator compared to determining the movement trajectory of the navigation indicator based on the position change of the first part determined from the continuous images. The number of frames between the two spaced images may be predetermined or dynamically adjusted. Exemplarily, the change in position of the first part in a plurality of frames in chronological order (e.g., a plurality of consecutive frames), or the coordinates of the navigation indicator transformed by the change in position, can be fitted to a smooth curve from which the trajectory of the navigation indicator can be determined.

In some embodiments, the camera is an RGB camera. The method <NUM> further includes a color space preprocessing step, which performs HSV color space processing on image data, so as to convert a color space of the image data into an HSV color space. The RGB camera usually uses three independent CCD sensors to obtain three color signals, which can capture very accurate color images and the accuracy of extraction and recognition of gesture features of the second part and the key point features of the first part can be improved. However, since RGB mode images are not conducive to skin color segmentation, according to the embodiment of the disclosure, color space preprocessing is further performed on the image data captured by the camera, the color space of the image data is converted into the HSV color space, so that the subsequent recognition and extraction of the gesture features of the second part and the key point features of the first part can be more accurate.

In some embodiments, the first machine learning model comprises a convolutional neural networks (CNN) model. The method <NUM> further includes a step of binarization preprocessing which preforms binarization processing on the image data to obtain binarization image data and a step of white balance preprocessing which performs white balance processing on the image data. Convolutional neural networks are input-to-output mappings that learn mapping relationships between inputs and outputs without the need for any precise mathematical expressions between inputs and outputs, and can be trained by known patterns to have the ability to map between input-output pairs, which are used to recognize the displacement of two-dimensional shapes with high accuracy. Therefore, adopting the convolutional neural network model to obtain the position of the first part has high accuracy. Further, the binarization of the image makes it possible to significantly reduce the amount of data of the image data to highlight the gesture contour of the second part, while the white balance processing corrects the lighting conditions of the image data so that the subsequent identification and extraction of the gesture features of the second part and the key point features of the first part can be more accurate.

In some embodiments, step S103 further includes: a final movement trajectory of the navigation indicator is determined by adopting a filtering algorithm and an anti-shake algorithm based on the position information of the first part. The filtering algorithm may include a Kalman filtering algorithm, and the anti-shake algorithm may include a moving average algorithm. In the embodiment of the disclosure, the position change of the key point features of the first part or the navigation indicator coordinate change determined by the position change is processed by adopting the filtering algorithm and the anti-shake algorithm, so that the movement trajectory of the navigation indicator can be smoother, and jitter of the navigation indicator is prevented.

<FIG> illustrates a schematic diagram of a scene of controlling a far-field display device based on a control method provided by an embodiment of the disclosure. The far-field display device <NUM> has a camera <NUM> which is configured to capture an image within a certain region in front of the far-field display device <NUM>. According to the control method provided by one or more embodiments of the disclosure, a user (not shown) may wave wrist <NUM> to move a navigation indicator <NUM> displayed on the far field display device <NUM>, and performs a gesture with hand <NUM> to give a specific control command to a visual element <NUM> to which the navigation indicator <NUM> is directed.

Referring to <FIG>, which illustrates a flow chart of a control method <NUM> provided based on another embodiment of the disclosure, and the method <NUM> includes step S201 to step S206:.

Accordingly, <FIG> illustrates a schematic diagram of the structure of a control method device <NUM> provided in accordance with an embodiment of the disclosure. The device <NUM> comprises:.

According to the control method device provided by one or more embodiments of the disclosure, the movement trajectory of the navigation indicator is determined based on the position information of the first part, and the control command is determined based on the gesture information of the second part, so that determination of the control command is independent of determination of the position of the navigation indicator. On the one hand, the determination of the control command is based on static gesture information, while the determination of the position of the navigation indicator is based on dynamic position changes, thus facilitating the use of different characteristic algorithms to determine the above two processes respectively. Exemplarily, the determination of the control command may be based on the static gesture information, while the determination of the position of the navigation indicator is based on the dynamic position change information. Therefore, for the above two different calculation characteristics, the position information of the first part and the gesture information of the second part may be calculated by adopting computing modules with corresponding characteristics respectively, thereby improving the targeting of the information acquisition, the accuracy of the calculation and the utilization of the calculation resources. On the other hand, the determination of the control command and the determination of the position of the navigation indicator are based on different body parts of the user, so that these determination processes do not interfere with each other, especially the shape of the contour of the first part does not change with the gesture of the second part, which can avoid the change of the gesture affecting the movement of the navigation indicator, and thus can improve the recognition accuracy of the user command.

For the embodiment of the device, which basically corresponds to the method embodiment, it is sufficient to refer to the description of the method embodiment for the relevant parts. The embodiments of the device described above are merely schematic, where the modules illustrated as separate modules may or may not be separate. Some or all of these modules may be selected according to practical needs to achieve the purpose of this embodiment solution. It can be understood and implemented without creative work by a person of ordinary skill in the art.

In some embodiments, the obtaining unit <NUM> is further configured to obtain the position information of the first part of the user based on the image by means of a first computing module and obtain the gesture information of the second part of the user based on the image by means of a second computing module.

In some embodiments, the control command unit <NUM> is further configured to control a controlled element based on the gesture information of the second part if the gesture information of the second part conforms to a preset first gesture.

In some embodiments, the control command unit <NUM> is further configured to not control the controlled element based on the gesture information of the second part if the gesture information of the second part does not conform to the preset first gesture.

In some embodiments, the obtaining unit <NUM> further comprises:.

In some embodiments, the device <NUM> further includes scrolling unit, configured to scroll the visual element to which the navigation indicator is directed based on position information of the first part obtained based on at least two frames of target image.

In some embodiments, the scrolling unit further includes:.

According to one or more embodiments of the disclosure, the target image is an image whose gesture information conforms to the second gesture. According to the invention, by translating the change in position of the first part into a scrolling effect of the visual element when the gesture information conforms to the second gesture, the user can control the navigation indicator to scroll the visual element, thereby enhancing interaction efficiency. The second gesture may include one or more preset hand gestures.

In some embodiments, the scrolling sub-unit is further configured to determine whether the motion information of the first part meets a preset motion condition, and if yes, to determine a scrolling direction and a scrolling distance of the visual element based on the motion information of the first part.

In some embodiments, the second gesture comprises splaying five fingers of one hand apart. The scrolling commands usually require fast movement of the gesture, and with fast movement, splaying five fingers apart is easier to recognize than other gestures, thus improving recognition accuracy.

In some embodiments, the movement trajectory unit <NUM> is further configured to determine the movement trajectory of the navigation indicator based on the position information of the first part, if the gesture information of the second part conforms to a preset third gesture, The third gesture may include a plurality of preset hand gestures. In this embodiment, if the gesture information of the second part conforms to the preset third gesture, the movement trajectory of the navigation indicator is determined based on the position information of first part. For example, the navigation indicator is moved only based on the position of the first part when the hand conforming to the preset hand gesture, which prevent the user from unintentionally moving the first part to cause the navigation indicator to move by mistake.

In some embodiments, the movement trajectory unit <NUM> is further configured to determine the movement trajectory of the navigation indicator based on the position information of the first part obtained from spaced images. In order to prevent jitter of the navigation indicator caused by the inevitable shaking of the user when waving the first part, the movement trajectory of the navigation indicator can be determined based on the position information of the first part obtained from the spaced images, which can reduce jitter of the navigation indicator compared to determining the movement trajectory of the navigation indicator based on the position change of the first part determined from the continuous images. Exemplarily, the change in position of the first part in a plurality of frames in chronological order (e.g., a plurality of consecutive frames), or the coordinates of the navigation indicator transformed by the change in position, can be fitted to a smooth curve from which the trajectory of the navigation indicator can be determined.

In some embodiments, the camera is an RGB camera. The device <NUM> further includes a color space preprocessing unit configured to perform HSV color space processing on image data, so as to convert a color space of the image data into an HSV color space. The RGB camera usually uses three independent CCD sensors to obtain three color signals, which can capture very accurate color images and the accuracy of extraction and recognition of gesture features of the second part and the key point features of the first part can be improved. However, since RGB mode images are not conducive to skin color segmentation, according to the embodiment of the disclosure, color space preprocessing is further performed on the image data captured by the camera, the color space of the image data is converted into the HSV color space, so that the subsequent recognition and extraction of the gesture features of the second part and the key point features of the first part can be more accurate.

In some embodiments, the first machine learning model comprises a convolutional neural networks (CNN) model. The device <NUM> further includes a binarization and white balance preprocessing unit configured to preform binarization processing and white balance processing on the image data. Convolutional neural networks are input-to-output mappings that learn mapping relationships between inputs and outputs without the need for any precise mathematical expressions between inputs and outputs, and can be trained by known patterns to have the ability to map between input-output pairs, which are used to recognize the displacement of two-dimensional shapes with high accuracy. Therefore, adopting the convolutional neural network model to obtain the position of the first part has high accuracy. Further, the binarization of the image makes it possible to significantly reduce the amount of data of the image data to highlight the gesture contour of the second part, while the white balance processing corrects the lighting conditions of the image data so that the subsequent identification and extraction of the gesture features of the second part and the key point features of the first part can be more accurate.

In some embodiments, the movement trajectory unit <NUM> is further configured to determine a final movement trajectory of the navigation indicator by adopting a filtering algorithm and an anti-shake algorithm based on the position information of the first part. The filtering algorithm may include a Kalman filtering algorithm, and the anti-shake algorithm may include a moving average algorithm. In the embodiment of the disclosure, the position change of the key point features of the first part or the navigation indicator coordinate change determined by the position change is processed by adopting the filtering algorithm and the anti-shake algorithm, so that the movement trajectory of the navigation indicator can be smoother, and the jitter of the navigation indicator is prevented.

Correspondingly, the disclosure further provides a terminal that is not covered by the appended claims comprising:.

Correspondingly, the disclosure further provides a non-transitory computer storage medium storing computer-readable instructions to perform the foregoing control method when the computer-readable instructions are executed by a computing device.

Referring now to <FIG>, a structural schematic diagram of terminal equipment <NUM> suitable for implementing an embodiment of the disclosure is shown. The terminal equipment in the embodiment of the present disclosure can include, but is not limited to, mobile terminals such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a Pad, a portable media player (PMP) and a vehicle-mounted terminal (e.g., vehicle-mounted navigation terminal), and fixed terminals such as a digital TV and a desktop computer. The terminal equipment shown in <FIG> is only an example, and should not bring any restrictions on the functions and application scope of the embodiments of the present disclosure.

As shown in <FIG>, the terminal equipment <NUM> can comprise a processing device (e.g., central processing unit, graphics processor, etc.) <NUM>, which can perform various appropriate actions and processing according to a program stored in a read-only memory (ROM) <NUM> or a program loaded into a random access memory (RAM) <NUM> from a storage device <NUM>. In the RAM <NUM>, various programs and data required for the operation of the terminal equipment <NUM> are also stored. The processing device <NUM>, the ROM <NUM>, and the RAM <NUM> are connected through a bus <NUM>. An Input/Output (I/O) interface <NUM> is also connected to the bus <NUM>.

Generally, the following devices can be connected to the I/O interface <NUM>: an input device <NUM> such as a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer and a gyroscope; an output device <NUM> such as a liquid crystal display (LCD), a speaker and a vibrator; a storage device <NUM> such as a magnetic tape and a hard disk; and a communication device <NUM>. The communication device <NUM> can allow the terminal equipment <NUM> to perform wireless or wired communication with other equipment to exchange data. Although <FIG> shows the terminal equipment <NUM> with various devices, it should be understood that it is not required to implement or provide all the devices shown. More or fewer devices may alternatively be implemented or provided.

Particularly, according to the invention, the processes described above with reference to the flowcharts is implemented as computer software programs. For example, the embodiments of the disclosure comprise a computer program product comprising a computer program carried by a computer-readable medium, and the computer program contains program codes for executing the method shown in the flowcharts. In such embodiment, the computer program can be downloaded and installed from a network through the communication device <NUM>, or installed from the storage device <NUM>, or installed from the ROM <NUM>. When the computer program is executed by the processing device <NUM>, the above functions defined in the method of the embodiments of the disclosure are executed.

It should be noted that the above-mentioned computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or component, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to, an electrical connector with one or more wires, a portable computer disk, a hard disk, an RAM, an ROM, an electrically erasable programmable read only memory (EPROM) or flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or component. In the disclosure, the computer-readable signal medium can comprise a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program codes are carried. This propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium can send, propagate or transmit the program for use by or in connection with the instruction execution system, device or component. The program codes contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to electric wire, optical cable, radio frequency (RF) or any suitable combination of the above.

In some embodiments, the client and the server can use any currently known or future developed network protocols such as HTTP (Hyper Text Transfer Protocol) to communicate, and can communicate with any form or medium digital data communications (e.g., communications networks) interconnected. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer-readable medium can be included in the terminal equipment, and can also exist alone without being assembled into the terminal equipment.

The computer-readable medium stores one or more programs that upon execution by the terminal cause the terminal to: receive an image, obtain position information of a first part and gesture information of a second part of a user based on the image, determine a movement trajectory of a navigation indicator based on the position information of the first part, and determine a control command based on the gesture information of the second part, the control command being used for controlling a visual element to which the navigation indicator is directed.

Or, the computer-readable medium stores one or more programs that upon execution by the terminal cause the terminal to : receiving an image, obtaining position information of a first part and gesture information of a second part of a user based on the image, determining a controlled element to which a navigation indicator is directed based on the position information of the first part, and determine a control command based on gesture information of the second part, the control command being used for controlling the controlled element to which the navigation indicator is directed.

Computer program codes for performing the operations of the disclosure can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be completely or partially executed on a user computer, executed as an independent software package, partially executed on a user computer and partially executed on a remote computer, or completely executed on a remote computer or server. In a case involving a remote computer, the remote computer can be connected to a user computer through any kind of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., connected through the Internet using an Internet service provider).

The flowcharts and block diagrams in the drawings show the architectures, functions and operations of possible implementations of systems, methods and computer program products according to various embodiments of the disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or part of a code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can also occur in a different order from those noted in the drawings. For example, two consecutive blocks can actually be executed in substantially parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented with dedicated hardware-based systems that perform specified functions or actions, or can be implemented with combinations of dedicated hardware and computer instructions.

The modules or units described in the embodiments of the disclosure can be implemented by software or hardware. The name of a module or unit does not constitute a limitation to the module or unit itself under certain circumstances. For example, the obtaining unit can also be described as "a unit for obtaining position information of a first part and gesture information of a second part of a user based on the image".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and etc..

Claim 1:
A computer-implemented control method (<NUM>) for controlling a visual element (<NUM>), the method comprising:
receiving (S101) an image from a camera;
obtaining (S102) position information of a first part of a user and gesture information of a second part of a user based on the image;
determining (S103) a movement trajectory of a navigation indicator (<NUM>) based on the position information of the first part; and
determining (S104) a control command for controlling the visual element to which the navigation indicator is directed based on the gesture information of the second part;
wherein the first part comprises a wrist of an arm of the user and the second part comprises a hand of the arm of the user, the first part belonging to a different body part from the second part;
characterized in that the method further comprises, when the gesture information of the second part conforms to a predetermined gesture, translating a change of position of the first part into a scrolling effect of the visual element.