2D pointing indicator analysis

In one embodiment, a method includes identifying a face, of a meeting attendee pointing to a display screen, in a first two-dimensional image from a two-dimensional video, determining at least one dimension of the face in the first two-dimensional image, defining a rectangle in the first two-dimensional image, at least one first dimension of the rectangle being a function of the at least one dimension of the face, searching for an image of a pointing indicator in the rectangle resulting in finding the pointing indicator at a first position in the rectangle, and calculating a cursor position of a cursor on the display screen based on the first position. Related apparatus and methods are also described.

TECHNICAL FIELD

The present disclosure generally relates to two-dimensional (2D) analysis of pointing indicators.

BACKGROUND

During meetings, it is very common to have participants in a room comment on the contents of a presentation slide. Often, the participants are pointing to a specific line or figure that they are commenting on, but it can be difficult for the other meeting participants to see exactly where the person is pointing. Research and development has been performed in the area of pointing detection, but usually, pointing detection is regarded as a special case in a generic gesture control scheme, often requiring very complex solutions to build a three-dimensional (3D) representation of a scene. 3D solutions typically require additional sensors and/or additional cameras (stereoscopic/triangulating, with angles from small up to 90 degrees).

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

An embodiment of the present disclosure includes a method including identifying a face, of a meeting attendee pointing to a display screen, in a first two-dimensional image from a two-dimensional video, determining at least one dimension of the face in the first two-dimensional image, defining a rectangle in the first two-dimensional image, at least one first dimension of the rectangle being a function of the at least one dimension of the face, searching for an image of a pointing indicator in the rectangle resulting in finding the pointing indicator at a first position in the rectangle, and calculating a cursor position of a cursor on the display screen based on the first position.

DETAILED DESCRIPTION

Reference is now made toFIG. 1, which is a pictorial view of a cursor positioning system10constructed and operative in accordance with an embodiment of the present disclosure.FIG. 1shows a plurality of meeting attendees12attending a meeting in a conference room14. The meeting may be a teleconference or video conference with one or more other meeting locations or the meeting may be a stand-alone meeting among the meeting attendees12in the conference room14. The meeting includes presenting an exemplary content item16on a display screen18. A video camera20is shown inFIG. 1centrally located atop the display screen18. It will be appreciated that the video camera20may be disposed at other locations around the display screen18. It will be appreciated than one or more other video cameras may be disposed in the conference room14for use in a video conference. The video camera20is typically a two-dimensional video camera for capturing two-dimensional images as part of a two-dimensional video. Optionally, the camera20includes three-dimensional, depth capturing, capabilities. One of the meeting attendees12, a meeting attendee12-1, is shown pointing with a finger22to the display screen18. The camera20captures images of the meeting attendee12-1including the finger22. The cursor positioning system10calculates a cursor position of a cursor24on the display screen18based on one or more of the captured images and displays the cursor24on the display screen18over the content item16. As the finger22of the meeting attendee12-1is moved around, this movement is detected by the cursor positioning system10and new cursor positions are calculated and the cursor24is moved to the newly calculated positions on the display screen18over the content item16. The assumption is that if the cursor24is not placed absolutely correctly at first, the meeting attendee12-1naturally adjusts the position of a hand36so that the cursor24is moved to the correct place in a similar manner to mouse control by a computer user.

Reference is now made toFIGS. 2-5, which are pictorial views illustrating calculation of cursor positions in the system10ofFIG. 1.FIGS. 2-5show different images26of the meeting attendee12-1pointing with the finger22towards different positions on the display screen18. A face28of the meeting attendee12-1is identified in each of the images26. Face detection algorithms are well known and readily available on a large range of equipment, even on existing video-conferencing systems. Some care must be taken if the face detection algorithm is sensitive to hands covering parts of the detected face28as shown inFIG. 5. The cursor positioning system10detects and records the position and size (at least one dimension) of the face28. The face28is shown surrounded by a box30in the images26for the sake of illustration only. A rectangle32is defined and is also shown in the images26for the sake of illustration only. The rectangle32defines a bounding box which likely includes the hand36with the finger22of the meeting attendee12-1. The position and dimensions of the rectangle32within each of the images26are generally based on one or more of the following: one or more dimensions of the face28; one or more dimensions of the display screen18; a relative position of the face28with respect to the display screen18; and a field of view of the camera20(FIG. 1) as will be described in more detail with reference toFIGS. 6A-7B.

The cursor positioning system10searches for the hand36with the pointing finger22in the rectangle32of each of the images26. A sliding window detection is used to search for the hand36with the pointing finger22within the rectangle32of each image26using an object recognition method for example, but not limited to, a neural network object recognition system. Hands are known to have a large variation of size, shape, color etc. and different people point differently. In order to provide accurate results, the neural network receives input of enough images of pointing hands, non-pointing hands and other images from the conference room14(FIG. 1) such as faces, clothes, chairs, computers etc.) to train the neural network. The size of the sliding window may be sized according to an expected size of the hand36with the pointing finger22. It will be appreciated that expected size of the hand36with the pointing finger22may be based on a size of the detected face28. The sliding window is moved across the rectangle32until the hand36with the pointing finger22is found by the object recognition system in the sliding window. Alternatively, the search for the image of the pointing finger22may be performed without using a sliding window, based on any other suitable image recognition technique for example, but not limited to, Scale-invariant feature transform (SIFT).

When the hand36with the pointing finger22is found in the rectangle32, the position of the hand is used to determine a corresponding cursor position of the cursor24over the content item16on the display screen18. It will be noted that the position of the cursor24on the display screen18is a horizontal flip of the position of the hand36with the pointing finger22found in the rectangle32with appropriate scaling to take into account the difference in sizes between the rectangle32and the display screen18. As the detected face28moves, the rectangle32may be moved correspondingly. When the meeting is part of a video conference, the cursor position is generally transmitted to the remote video equipment as well, either in encoded video or through a parallel communication channel for display on display device(s) in the remote locations.

It should be noted that neural network object recognition may not be performed on each of the images26captured by the camera20(FIG. 1). Neural network object recognition may be performed periodically, for example, but not limited to, every 100 milliseconds or every one or several seconds. An object tracking technique, such as edge detection, may be used to detect movement of the detected hand36with the pointing finger22between detections by the neural network object recognition process. Combining the neural network object recognition with object tracking may result in a quicker cursor movement than the using neural network object recognition alone. In any event, the presence of the hand36with the pointing finger22may be reconfirmed periodically using the neural network object recognition. Whenever the neural network object recognition no longer detects the hand36with the pointing finger22, the cursor24is typically removed from the display screen18.

It should be noted that the cursor positioning system10does not try to find the exact point on the display screen18that the meeting attendee12-1is pointing to. The cursor positioning system10generally does not take into account the direction of the finger22of the meeting attendee12-1, or the direction of the hand36or an arm38, but rather how the face28and the hand36are positioned relative to the camera20. As described above, the assumption is that if the cursor24is not placed absolutely correctly at first, the meeting attendee12-1naturally adjusts the position of the hand36so that the cursor24is moved to the correct place in a similar manner to mouse control by a computer user.

Three methods for estimating the size and position of the rectangle32in each of the images26are now described. The methods discuss calculating a height46and a width62of the rectangle32. The first method is described with reference toFIGS. 6A-C, the second method is described with reference toFIG. 7and the third method is described after the second method.

Reference is now made toFIG. 6A, which is a side view of the meeting attendee12-1pointing at the display screen18. The cursor positioning system10is operative to provide an estimation of a distance (D)42from the display screen18to the meeting attendee12-1based on an assumption about the average length (F)40of a human adult face and an angular height (A)43of the face28in the image26(FIGS. 2-5). The angular height (A)43of the face28in the image26may be determined from the height of the face28in the image26and knowledge of the field of view of the camera20. For example, if the height of the face28occupies 6% of the image26and the field of view of the camera20is 90 degrees, then the angular height (A)43of the face28is 5.4 degrees. Assuming minor errors for small angles, the distance (D)42may be calculated by:

The average height of the human adult face from the menton to the crinion according to one study is between 18 to 19 centimeters (cm). By way of example, assuming the face length (F)40is 18 cm and angular height (A)43is 5.4 degrees, the distance (D)42is 190 cm. It will be appreciated that other measurements of the face28may be used in the calculating the distance (D)42(and any of the other distances described herein), for example, but not limited to, the distance from the stomion to the top of the head.

Reference is now made toFIG. 6B.FIG. 6Bshows the finger22pointing to the top (solid line used for arm38) and pointing to the bottom (dotted line used for arm38) of the display screen18. Lines45show the line of sight from an eye47(or eyes47) of the meeting attendee12-1to the top and bottom of the display screen18. It can be seen that a ratio between the height (RH)46and an estimated length (L)44of the arm38is equal to a ratio between a known height (H)48of the display screen18and the distance (D)42between the display screen18and the face28. The above ratios may be expressed as follows:

which may be rearranged as,

Therefore, the height (RH)46of the rectangle32may be estimated based on the estimated length (L)44of the arm38, the known height (H)48of the display screen18and the estimated distance (D)42between the display screen18screen and the face28. By way of example, assuming a typical arm length of 60 cm, a screen height of 70 cm, and a distance (D)42of 190 cm, the height (RH)46of the rectangle32is 22 cm. It will be appreciated that the length44of the arm38may alternatively be estimated based on image recognition and analysis of the arm38in the image26. It will be appreciated that the estimated height (RH)46may be over estimated by a certain value, for example, but not limited to, 10% or 25%, or any other suitable value, so that the height (RH)46ensures that the rectangle32is tall enough to encompass the high and low positions of the hand36pointing at the display screen18and also to take into account that the various distances and positions discussed above are generally based on estimations and assumptions about the human body. If the height46is over-estimated too much then some of the meeting attendees12may be unable to reach the corners of the rectangle32which correspond to moving the cursor24(FIGS. 2-5) to the corners of the display screen18.

Reference is now made toFIG. 6C. An angular size (B)49of the height46in the image26(FIG. 5) may be determined using the following formula.

Using the exemplary dimensions used inFIGS. 6A and 6Bgives an angular size (B)49of 9.6 degrees. An angular width of the rectangle32(FIG. 5) may be estimated using the angular size (B)49and an aspect ratio of the display screen18. So for example, if the screen has a height of 70 cm and a width of 105 cm, then the angular width of the rectangle32in the image26(FIG. 5) will be:

Alternatively, an angular width may be estimated using other geometric calculations described in more detail with reference toFIG. 7.

Positioning of the rectangle32(FIG. 5) in the image26(FIG. 5) is now discussed. The top of the rectangle32may be disposed at the level of the eyes47which may be assumed to be half way up the box around the face28. It will be appreciated that this may need some individual adjustment based on the specific face detection implementation. Additionally, an average user would probably hold his/her hand slightly lower than on the direct line between the eye47and the point being pointed to and therefore the top of the rectangle32may be lower than the level of the eyes47. User testing may need to be performed to determine the most natural position of the rectangle32.

It may be assumed that horizontal positioning of the rectangle32is such that a center of the rectangle32is centered horizontally with the face28. Accuracy may be improved for meeting attendees12sitting off-axis from the display screen18and the camera20, so that the rectangle32is shifted more to one side of the face28. Adjustments for off-axis positioning may be determined based on practical user testing in the conference room14(FIG. 1).

Another factor with off-axis sitting is that faces will have the same height but are generally narrower than centrally sitting meeting attendees12. In such a case, accuracy of the rectangle32may be improved by making the rectangle32narrower, probably by a factor close to cosine(alpha) where alpha is the angle to the face28from the camera20center line.

The height and width of the rectangle32may be estimated using the above mentioned method for future calculations of the dimensions of the rectangle32. Alternatively, as it is known that the height46of the rectangle32is 9.6/5.4=1.78 times the length of the face28in the image26(FIGS. 2-5) in the above mentioned example, it may be assumed that during future calculations that the height46is 1.78 times the length of the face28in the conference room14(FIG. 1) with the current set up of the display screen18and the camera20.

Reference is now made toFIG. 7, which is a plan view of the meeting attendee12-1pointing at the display screen18illustrating a method of calculation of a horizontal dimension for use in the system10ofFIG. 1.FIG. 7shows the finger22pointing to the left (solid line used for arm38) and the right of the display screen18(dotted line used for arm38). Lines51show the line of sight from the eyes47of the meeting attendee12-1to the left and right of the display screen18. It can be seen that a ratio between the width (RW)62and an estimated length (L)44of the arm38is equal to a ratio between a known width (H)56of the display screen18and the distance (D)42between the display screen18screen and the face28. The above ratios may be expressed as follows:

which may be rearranged as,

Therefore, the width (RW)62of the rectangle32(FIG. 5) may be estimated based on the estimated length (L)44of the arm38, the known width (W)56of the display screen18and the estimated distance (D)42between the display screen18and the face28(for example calculated using the method described with reference toFIG. 6A). By way of example, assuming a typical arm length L of 60 cm, a screen width W of 105 cm, and a distance (D)42of 190 cm, the width (RW)62of the rectangle is calculated as 33 cm.

An angular size (C)53of the width62in the image26(FIG. 5) may be determined using the following formula:

Using the exemplary dimensions above, the angular size (C)53of the width of the rectangle32is calculated as 14.4 degrees.

It will be appreciated that either the angular size B or angular size C may be calculated using the methods described above with reference toFIGS. 6A-CandFIG. 7, and the other angular size C or B may be calculated based on the known aspect ratio of the display screen18, respectively. It will be appreciated that both the angular size B and the angular size C may be calculated using the methods described above with reference toFIGS. 6A-CandFIG. 7, respectively.

It will be appreciated that the estimated width62may be over estimated by a certain value, for example, but not limited to, 10% or 25%, or any other suitable value, so that the width62ensures that the rectangle32is wide enough to encompass the hand36pointing at the left and the right edges of the display screen18. If the width62is over-estimated too much then some of the meeting attendees12may be unable to reach the corners of the rectangle32which correspond to moving the cursor24(FIGS. 2-5) to the corners of the display screen18.

A simplified method for calculating the dimensions of the rectangle32(FIG. 1) may be based on assuming the width and/or height of the rectangle32are certain multiples of the face width and/or height (or other dimension of the face28). The multiples used in the calculation may, or may not, be based on configuration testing of the cursor positioning system10(FIG. 1) in the conference room14(FIG. 1), for example by positioning the meeting attendee12-1at one or more positions in the conference room14with the meeting attendee12-1pointing to the top/bottom and/or left/right of the display screen18and measure the distance between the fingers22of the meeting attendee12-1at the various positions to give the dimension(s) of the rectangle32.

Reference is now made toFIG. 8, which is a partly pictorial, partly block diagram view of a collaboration server78used in calculating a cursor position in the system10ofFIG. 1. The collaboration server78may be operative to establish and execute collaboration events between different video end-points (VEPs)80via a network82. A video end-point is typically video and audio equipment for capturing and transferring video and audio to one or more other VEPs in other locations and receiving audio and video from one or more VEPs in other locations for rendering in the current location. The collaboration server78may also be operative to process collaboration event data such as calculating the cursor position of the cursor24on the display screen18included in one of the video end-points80. It will be appreciated that the cursor position may be calculated for display on the display screen18without transmitting the cursor position and/or a presentation including the cursor24to any VEP in other locations, for example, but not limited to, when a video conference is not in process and the display screen18and camera20are being used to display presentation content locally to the meeting attendees12in the conference room14(FIG. 1) and not to meeting attendees12in other locations. The collaboration server78may include a processor84, a memory86, a data bus88, a storage unit90and one or more interfaces92. The memory86is operative to store data used by the processor84. The data bus88is operative to connect the various elements of the collaboration server78for data transfer purposes. The storage unit90is operative to store various data including collaboration event data and other data used by the cursor positioning system10. The interface(s)92are used to transfer data between the collaboration server78and the video end-points80.

Reference is now made toFIG. 9, which is a partly pictorial, partly block diagram view of a device94used in calculating a cursor position in the system10ofFIG. 1in accordance with an alternative embodiment of the present disclosure. The device94may be disposed in the location of the conference room14(FIG. 1) where the display screen18and the camera20are located. The device94includes a processor96, a memory98, a data bus100, a storage unit102, and one or more interfaces104. The memory98is operative to store data used by the processor96. The data bus100is operative to connect the various elements of the device94for data transfer purposes. The storage unit102is operative to store various data including data used by the cursor positioning system10. The interface(s)104are used to transfer data between the device94and the collaboration server78and the video end-points80via the network82.

Reference is now made toFIG. 10, which is a diagram illustrating machine learning setup for use in the system10ofFIG. 1. A plurality of images108of a hand with a pointing finger (shown) and hands without a pointing finger (not shown) and other images (not shown) from the conference room14(FIG. 1) such as faces, clothes, chairs, computers are collected (block110). If other pointing indicators, for example, but not limited to, a hand holding a pen or a ruler, and/or part of a hand with a pointing finger, and/or part of a hand holding a pen or a ruler are to be used to point with, images of other pointing indicators may be used as well. The images108are input into a machine learning algorithm so that the machine learning algorithm can learn to find a hand with a pointing finger in an image (block112).

Reference is now made toFIG. 11, which is a flow chart showing exemplary steps in a method114of calculating a cursor position in the system10ofFIG. 1. The method114is described by way of the processor96ofFIG. 9. It will be appreciated that the processor84(FIG. 8) may be used to perform one or more of the steps described below as being performed by the processor96.

The processor96is operative to analyze one of the images26(FIGS. 2-5) and identify the face28, of the meeting attendee12-1pointing to the display screen18, in that image26(block116). The image26is generally a two-dimensional image from a two-dimensional video. The step of block116is typically triggered by the content item16(FIG. 1) being shared on the display screen18(FIG. 1). If there is more than one face in that image26, the processor96may be operative to find the talking face in that image26on which to base the definition of the rectangle32(FIGS. 2-5) or alternatively define the rectangle32for each face in the image26which may lead to more than one cursor24on the display screen18, one cursor24per pointing finger. The processor96is operative to determine at least one dimension (e.g., a height and/or other dimension(s)) of the face28(FIGS. 2-5) in the image26(block118). The processor96is operative to define the rectangle32in the images26(block120). The step of block120includes two sub-steps, the steps of blocks122and124which are now described. The processor96is operative to calculate the dimension(s) of the rectangle32as a function of: the dimension(s) of the face28(identified in the image26); optionally knowledge about the field of view of the camera20(FIGS. 1-5); and the dimension(s) of the display screen18(FIGS. 6A-6C and 7) (block122). It should be noted that the aspect ratio of the rectangle32may be set to be the same as the aspect ratio of the display screen18. In such a case, if one of the dimensions of the rectangle32is calculated (as described above), the other dimension of the rectangle32may be determined so that the aspect ratio of the rectangle32is the same as the aspect ratio of the display screen18as described above, with reference toFIGS. 6A-Cand7. The processor96is operative to calculate a horizontal and vertical position of the rectangle32in the image26as described above, with reference toFIGS. 6A-Cand7(block124).

The processor96is operative to search for an image of a pointing indicator in the rectangle32(FIGS. 2-5) resulting in finding the pointing indicator at a first position in the rectangle32. The pointing indicator may be the hand36with the finger22described above with reference toFIGS. 2-5. Alternatively or additionally, the cursor positioning system10may be operative to search the rectangle32for other pointing indicators, for example, but not limited to, a hand holding a pen or a ruler, and/or part of a hand with a pointing finger, and/or part of a hand holding a pen or a ruler. The processor96is operative to search for the image of the pointing indicator in a sliding window which is moved around the rectangle32(block126). The search for the image of the pointing indicator may be based on machine learning of a plurality of images of pointing indicators. The size of the sliding window may be set as a function of one or more dimensions of the face28as discussed above with reference toFIGS. 2-5. Alternatively, the search for the image of the pointing indicator may be performed without using a sliding window, based on any other suitable image recognition technique for example, but not limited to, Scale-invariant feature transform (SIFT). The processor96is operative to calculate a cursor position of the cursor24(FIG. 2-5) on the display screen18based on the first position (block128). The processor96may be operative to calculate the cursor position on the display screen18based on a horizontal flip and scaling of the first position of the pointing indicator in the rectangle32.

The processor96is operative to prepare a user interface screen presentation132(FIGS. 2-5) including the cursor24(FIGS. 2-5) placed at the calculated cursor position (block130). The processor96is operative to output the user interface screen presentation132for display on the display screen18(FIGS. 2-5) (block134).

The processor96is optionally operative to return via branch136to the step of block126to resume searching for the pointing indicator in the rectangle32. The processor96is optionally operative to return periodically (for example, but not limited to, every 500 milliseconds or every few seconds) via branch138from the step of block134to the step of block116to identify the face28in a new image26and continue the processing described above from the step of block116. In other words, the process of the method114after the step of block132may follow the processing of branch136and periodically follow the processing of the branch138.

In accordance with an alternative embodiment, instead of the method114following the branches136,138, the processor96may be operative to track movement of the pointing indicator over a plurality of the images26(FIGS. 2-5) using an object tracking method such as edge tracking as described above with reference toFIGS. 2-5(block140) and to return periodically (for example, but not limited to, every 1 to 10 seconds) via branch142to the step of block116and then proceeding to search for the image of the pointing indicator using a sliding window in different images from the two-dimensional video.

In accordance with yet another alternative embodiment, instead of the method114following the branches136,138, the processor96may be operative to track movement of the pointing indicator over a plurality of the images26(FIGS. 2-5) using an object tracking method such as edge tracking as described above with reference toFIGS. 2-5(block140) and to return periodically (for example, but not limited to, every 500 milliseconds or every few seconds) via branch144to the step of block126(and continue the processing described above from the step of block126) and return from the block140less frequently (than the flow down the branch144) (for example, but not limited to, every 1 to 10 seconds) via branch142to the step of block116.

The processor96is operative to remove the cursor from the user interface screen presentation132(FIGS. 2-5) when the pointing indicator is not found in one of the two-dimensional images26(FIGS. 2-5) from the two-dimensional video.

In practice, some or all of these functions may be combined in a single physical component or, alternatively, implemented using multiple physical components for example, graphical processing unit(s) (GPU(s)) and/or field-programmable gate array(s) (FPGA(s)). These physical components may comprise hard-wired or programmable devices, or a combination of the two. In some embodiments, at least some of the functions of the processing circuitry may be carried out by a programmable processor under the control of suitable software. This software may be downloaded to a device in electronic form, over a network, for example. Alternatively or additionally, the software may be stored in tangible, non-transitory computer-readable storage media, such as optical, magnetic, or electronic memory.

It is appreciated that software components may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques. It is further appreciated that the software components may be instantiated, for example: as a computer program product or on a tangible medium. In some cases, it may be possible to instantiate the software components as a signal interpretable by an appropriate computer, although such an instantiation may be excluded in certain embodiments of the present disclosure.

It will be appreciated by persons skilled in the art that the present disclosure is not limited by what has been particularly shown and described hereinabove. Rather the scope of the disclosure is defined by the appended claims and equivalents thereof.