Patent Description:
Motion tracking systems are often used in a variety of applications, including, for example, in the medical field, in the movie and video game industries, and the like. There remains a continued need for new systems and methods are needed to accurately track the motion of an object in all directions using a device, such as a smartphone, with limited processing power. The present disclosure addresses these and other problems.

"<NPL> et al. refers to a main idea to apply a 3D colored target at the end of a toothbrush and to track and analyze its motion.

<CIT> refers to a special marker fixed to the end of a toothbrush to allow a processing device, by means of a video camera, to locate a spatial position of the toothbrush, and also to analyse a direction of motion.

"<NPL> et al. refers to a vision-based brushing tracker which consists of a box-shaped brush extension and a web camera. The box-shaped brush extension is provided with four rectangular faces. Each face is coded with LED markers showing unique patterns distinguishable for computer vision. A stem on the brush extension is affixable with VELCRO to the end of a standard toothbrush. The web camera is positioned above a user to capture an overhead view of a toothbrush extension movement. The brushing tracker uses the web camera to track a LED-coded toothbrush extension.

<CIT> refers to a through-the-lens tracking system/method which when in a PREVIZION system (or method) enables sub-pixel camera tracking.

<CIT> refers to an oral hygiene device containing a handle and a head, where either or both are adapted to include a pattern for visual detection of movement and orientation by an associated camera. The pattern is positionable at a location on the handle.

<CIT> refers to a toothbrush which includes a brush body and a brush end. A target, in the form of a retroreflector, a tag, color or a combination thereof, is couplable or integral with the brush body and/or the brush end.

The present invention provides a method for estimating a pose of an oral hygiene device relative to a location, as defined in claim <NUM>, and a motion tracking system as defined in claim <NUM>.

According to some implementations of the present disclosure, a method for estimating a pose of an oral hygiene device including a pattern and a plurality of groups of visual markers relative to a location the method includes receiving image data reproducible as an image of at least a portion of the oral hygiene device. The method also includes analyzing, using one or more processors, the image data to identify a region of interest within the image, the region of interest including at least a portion of the pattern therein. A method not forming an embodiment of the invention further includes: identifying, using at least one of the one or more processors, all candidate visual markers within the region of interest, obtaining a first proposed three-dimensional pose of the oral hygiene device, validating the first proposed three-dimensional pose of the oral hygiene device, and obtaining a second proposed three-dimensional pose of the oral hygiene device based on the validated first proposed three-dimensional pose.

The method according to the present disclosure, which is a method for estimating a pose of an oral hygiene device including a pattern and a plurality of groups of visual markers relative to a location, includes: (a) receiving image data reproducible as an image of at least a portion of the oral hygiene device; (b) analyzing, using one or more processors, the image data to identify a region of interest within the image, the region of interest including at least a portion of the pattern therein; (c) responsive to identifying the region of interest, segmenting, using at least one of the one or more processors, the region of interest into a plurality of sub-regions, each of the plurality of sub-regions being defined by a plurality of pixels having a common color; (d) identifying, using at least one of the one or more processors, all candidate visual markers within the region of interest; (e) creating a plurality of distinct sets of the candidate visual markers; (f) selecting a first one of the plurality of distinct sets of the candidate visual markers; (g) selecting a first one of a plurality of distinct sets of model markers associated with a three-dimensional model of the oral hygiene device; (h) evaluating the selected set of the candidate visual markers and the selected set of model markers using a perspective-three-point algorithm to obtain a proposed three-dimensional pose of the oral hygiene device; (i) based on the proposed three-dimensional pose of the oral hygiene device, predicting a position within the region of interest for a predetermined number of the candidate visual markers; (j) comparing the predicted positions for the predetermined number of the candidate visual markers with actual positions of all of the candidate visual markers within the region of interest; (k) responsive to a determination that at least a substantial portion of the predicted positions correspond with the actual positions, validating the proposed three-dimensional pose; and (I) responsive to a determination that less than the substantial portion of the predicted positions correspond with the actual positions, repeating steps (f) - (k).

According to other implementations of the present disclosure, a motion tracking system includes an oral hygiene device, a tracking element, a camera, one or more processors, and a memory device. The oral hygiene device includes a head and a handle. The tracking element is coupled to the oral hygiene device and includes a pattern and a plurality of groups of visual markers. The memory device stores instructions that, when executed by at least one of the one or more processors cause the motion tracking system to: capture, using the camera, an image of at least a portion of the oral hygiene device; analyze, using at least one of the one or more processors, the image to identify a region of interest within the image, the region of interest including at least a portion of the pattern of the tracking element therein; identify, using at least one of the one or more processors, all candidate visual markers within the region of interest; create a plurality of distinct sets of the candidate visual markers; select a first one of the plurality of distinct sets of the candidate visual markers; select a first one of a plurality of distinct sets of model markers associated with a three-dimensional model of the oral hygiene device stored in the memory device; evaluate the selected set of the candidate visual markers and the selected set of model markers using a perspective-three-point algorithm to obtain a proposed three-dimensional pose of the oral hygiene device; based on the proposed three-dimensional pose of the oral hygiene device, predict a position within the region of interest for a predetermined number of the candidate visual markers; compare the predicted positions for the predetermined number of the candidate visual markers with actual positions of all of the candidate visual markers within the region of interest; and responsive to a determination that at least a substantial portion of the predicted positions correspond with the actual positions, validate the proposed three-dimensional pose.

According to one implementation not forming an embodiment of the invention, a motion tracking element configured to be coupled to an oral hygiene device includes a body, a pattern on an outer surface of the body, and a plurality of groups of visual markers on the outer surface of the body.

The above summary of the present disclosure is not intended to represent each embodiment, or every aspect, of the present disclosure. Additional features and benefits of the present disclosure are apparent from the detailed description and figures set forth below.

While the disclosure is susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and are described in detail herein. It should be understood, however, that the disclosure is not intended to be limited to the particular forms disclosed.

Referring to <FIG>, a motion tracking system <NUM> includes an oral hygiene device <NUM>, a tracking element <NUM>, a camera <NUM>, a processor <NUM>, and a memory device <NUM>. The motion tracking system <NUM> is generally used to estimate a pose of the oral hygiene device <NUM> in a three-dimensional space relative to a location, such as, for example, the camera <NUM>.

The oral hygiene device <NUM> includes a head <NUM> and a handle <NUM>. The head <NUM> is coupled to a first end of the handle <NUM> and includes a plurality of bristles for brushing teeth. The head <NUM> and the handle <NUM> can be unitary or monolithic or, alternatively, the head <NUM> can be removably coupled to the handle <NUM> such that the handle <NUM> is interchangeable (e.g., with a replacement head). The handle <NUM> has a generally cylindrical shape, but more generally can be any suitable size and shape. The handle <NUM> can include an ergonomic grip to aid a user in gripping the handle <NUM>. The oral hygiene device <NUM> can include an electric motor (not shown) to vibrate and/or oscillate or otherwise provide motion to the head <NUM> to aid in brushing teeth. More generally, the oral hygiene device <NUM> can be any manual toothbrush or electric toothbrush.

The tracking element <NUM> can be detachably coupled (directly or indirectly) to, fixedly or rigidly coupled (directly or indirectly) to, or formed integrally with, the handle <NUM> of the oral hygiene device <NUM>. Further, the tracking element <NUM> can be coupled to the handle <NUM> of the oral hygiene device <NUM> such that an axis of the tracking element <NUM> corresponds with or is co-axial with an axis of the handle <NUM>. The tracking element <NUM> includes a pattern <NUM> and a plurality of visual markers <NUM>. The tracking element <NUM> is generally made from a flexible material. For example, the tracking element <NUM> can be made from a non-conductive material such as, for example, a rubber or elastomer material, a polymer material, or any combination thereof.

The camera <NUM> is a digital camera that is generally used to capture still images, video images, or both, of at least a portion of the oral hygiene device <NUM> and the tracking element <NUM>. Typically, the oral hygiene device <NUM> is positioned between the user <NUM> and the camera <NUM> such that the field of view of the camera <NUM> encompasses at least a portion of the oral hygiene device <NUM> and at least a portion of the tracking element <NUM>.

The processor <NUM> is communicatively coupled to the camera <NUM> and the memory device <NUM>. The processor <NUM> executes instructions (e.g., an associated application) stored in the memory device <NUM> to control the various components of the system <NUM> to which it is communicatively coupled.

In some implementations, the system <NUM> further includes a housing <NUM>. In such implementations, the camera <NUM>, the processor <NUM>, the memory device <NUM>, or any combination thereof can be integrated in the housing <NUM>. For example, the housing <NUM> can be a smartphone. Alternatively, some or all of the various components can be decoupled from one another, and some can be included in a base station (not shown) for the oral hygiene device <NUM>.

Referring to <FIG>, an oral hygiene device <NUM> that is the same as or similar to the oral hygiene device <NUM> is coupled to a tracking element <NUM> that is the same as or similar to the tracking element <NUM> described above.

The oral hygiene device <NUM> includes a head (not shown) and a handle <NUM>. The head is a coupled to a first end of the handle <NUM> and the tracking element <NUM> is coupled to a second end of the handle <NUM> that is opposite the head.

The tracking element <NUM> includes an upper portion <NUM>, a lower portion <NUM>, a pattern <NUM>, a first group of visual markers <NUM>, a second group of visual markers <NUM>, and a third group of visual markers <NUM>. The upper portion <NUM> has a generally cylindrical configuration and includes a cavity <NUM>. The cavity <NUM> is sized and shaped to receive the handle <NUM> of the oral hygiene device <NUM> therein to removably couple the tracking element <NUM> to the handle <NUM> using a press or interference fit. The upper portion <NUM> may, for example, be formed from an elastomeric material in the form of a sleeve and be configured and arranged to receive and conform to the second end of the handle <NUM>. Alternatively, the tracking element <NUM> can be coupled to the handle <NUM>. using other mechanisms, such as, for example, a threaded connection, an adhesive connection, a hook and loop fastener, a tab and aperture system, a press or interference fit connection, a snap fit connection, a force fit connection, a twist-lock connection, or the like, or any combination thereof. Alternatively, in some implementations, the tracking element <NUM> includes a male attachment feature and the second end of the handle <NUM> can include a cavity that is similar to the cavity <NUM> and is sized and shaped to receive at least a portion of the male attachment feature therein. In such implementations, the tracking element <NUM> male attachment feature and the cavity of the handle <NUM> can be coupled using any of the fastening mechanisms described above. Advantageously, in this configuration, the tracking element <NUM> can be removed from the oral hygiene device <NUM> if the user does not desire to use the tracking element <NUM> during a given brushing session. Further, the tracking element <NUM> can be removed from the oral hygiene device <NUM> and coupled to a second oral hygiene device such as, for example, when the user replaces the oral hygiene device <NUM> at the end of its useful life or in the event another user desires to use the tracking element <NUM> on another oral hygiene device. Alternatively, the tracking element <NUM> and the handle <NUM> can be unitary and/or monolithic.

As shown in <FIG>, the lower portion <NUM> of the tracking element <NUM> can have a generally spherical configuration, although other sizes and shapes are possible. As shown, the pattern <NUM> formed on an outer surface thereof and is flush with the outer surface of the lower portion <NUM> and includes a background <NUM> and a plurality of indicators <NUM>.

As shown, each of the plurality of indicators <NUM> have an amorphous shape that is generally circular-like or oval-like. The shape of the plurality of indicators <NUM> shown in <FIG> is preferable because this shape minimizes blur associated with movement of the oral hygiene device <NUM> and the tracking element <NUM> when capturing an image of the same. Alternatively, one or more of the indicators of the plurality of indicators <NUM> can have a generally triangular shape, a generally rectangular shape, a polygonal shape, or any combination thereof. While each of the plurality of indicators <NUM> is shown as having the same shape and size, each of the plurality of indicators <NUM> can have a different size or substantially the same size (e.g., diameter).

Each of the plurality of indicators <NUM> have a first color and the background <NUM> has a second color that is different than the first color. In one example, the background <NUM> is an orange color and each of the plurality of indicators <NUM> are a black color. Alternatively, the background <NUM> can be a black color and each of the plurality of indicators <NUM> can be a generally orange color, although other colors for the background <NUM> and the plurality of indicators <NUM> are possible (e.g., red, green, blue, yellow, orange, purple, etc.). In general, a high contrast between the color of the background <NUM> and each of the plurality of indicators <NUM> is preferable so as to clearly define each of the plurality of indicators <NUM>. The plurality of indicators <NUM> defining the pattern <NUM> on the tracking element <NUM> can have between about ten indicators and about one hundred indicators, between about twenty indicators and about sixty indicators, between about thirty-five and about forty-five indicators, or any suitable number of indicators.

The plurality of indicators <NUM> and the background <NUM> of the pattern <NUM> can be formed using a variety of mechanisms. For example, at least some of the plurality of indicators <NUM> and/or the background <NUM> of the pattern <NUM> can be printed or embossed on the outer surface of the tracking element <NUM>. Alternatively, at least some of the plurality of indicators <NUM> and/or the background <NUM> can be integral with the lower portion <NUM>.

Referring to <FIG>, the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> are coupled to the outer surface of the lower portion <NUM> and protrude therefrom. As shown, each visual marker in the groups of visual markers <NUM>, <NUM>, <NUM> has a generally ovoid, dome-like or semi-spherical shape. The first, second, and third groups of visual markers <NUM>, <NUM>, <NUM> can be coupled to the outer surface of the lower portion <NUM> using an adhesive connection, for example, or more generally any other suitable mechanism. Alternatively, each of the visual markers can be unitary and/or monolithic with the lower portion <NUM> of the tracking element <NUM>.

Referring to <FIG>, the first group of visual markers <NUM> includes a first visual marker 241a, a second visual marker 241b, a third visual marker 241c, and a fourth visual marker 241d. The handle <NUM> has a front surface 214a and a rear surface 214b. The cleaning elements (e.g., bristles) on the head (not shown) extend from the front surface 214a of the handle <NUM>. To illustrate all of the visual markers, <FIG> is a front view of the oral hygiene device <NUM> (i.e., includes the front surface 214a) and <FIG> is a rear view the oral hygiene device <NUM> (i.e., includes the rear surface 214b).

In some implementations, the first group of visual markers <NUM> extends along a first circumferential length of the lower portion <NUM> that is proximate to the upper portion <NUM> and the handle <NUM>. As shown, each of the visual markers in the first group of visual markers <NUM> are evenly spaced from one another along the first circumferential length.

The second group of visual markers <NUM> includes a first visual marker 242a, a second visual marker 242b, a third visual marker 242c, and a fourth visual marker 242d. The second group of visual markers <NUM> extends along a second circumferential length of the lower portion <NUM> that is spaced from the first circumferential length. As shown, each of the visual markers in the second group of visual markers <NUM> are evenly spaced from one another along the first circumferential length, but offset circumferentially from the visual markers in the first group of visual markers <NUM>. Specifically, the first visual marker 242a is positioned between the first visual marker 241a and the second visual marker 241b of the first group of visual markers <NUM> (<FIG>), the second visual marker 242b is positioned between the second visual marker 241b and the third visual marker 241c of the first group of visual markers <NUM> (<FIG>), the third visual marker 242c is positioned between the third visual marker 241c and the fourth visual marker 241d of the first group of visual markers <NUM> (<FIG>), and the fourth visual marker 242d is positioned between the fourth visual marker 241d and the first visual marker 241a of the first group of visual markers <NUM> (FIG.

The third group of visual markers <NUM> includes a first visual marker 243a, a second visual marker 243b, a third visual marker 243c, and a fourth visual marker 243d. The third group of visual markers <NUM> extends along a third circumferential length of the lower portion <NUM> that is spaced from the second circumferential length and distal to the upper portion <NUM> and the handle <NUM>. The first circumferential length, the second circumferential length, and the third circumferential length are evenly spaced from one another such that the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> are evenly spaced from one another.

The third group of visual markers <NUM> includes a first visual marker 243a, a second visual marker 243b, a third visual marker 243c, and a fourth visual marker 243d. The first visual marker 243a is aligned with the first visual marker 241a of the first group of visual markers <NUM> such that the first visual marker 243a is positioned between the fourth visual marker 242d and the first visual marker 242a of the second group of visual markers <NUM>. The second visual marker 243b is aligned with the second visual marker 241b of the first group of visual markers <NUM> such that the second visual marker 243b is positioned between the first visual marker 242a and the second visual marker 242b of the second group of visual markers <NUM>. The third visual marker 243c is aligned with the third visual marker 241c of the first group of visual markers <NUM> such that the third visual marker 243c is positioned between the second visual marker 242b and the third visual marker 242c of the second group of visual markers <NUM>. The fourth visual marker 243d is aligned with the fourth visual marker 241d of the first group of visual markers <NUM> such that the fourth visual marker 243d is positioned between the third visual marker 242c and the fourth visual marker 242d of the second group of visual markers <NUM>.

Each of the visual markers within the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> has a distinctive color. For example, in the first group of visual markers <NUM>, the first visual marker 241a has a first color, the second visual marker 241b has a second color, the third visual marker 241c has a third color, and the fourth visual marker 241d has a fourth color. The first color, the second color, the third color, and the fourth color are all different from one another. Preferably, the first color, second color, third color, and fourth color are separate and distinct colors that are spaced out along the color spectrum. For example, each color can be spaced from the other colors by between about a <NUM> wavelength to a <NUM> wavelength in the color spectrum, about a <NUM> wavelength in the color spectrum to about a <NUM> wavelength in the color spectrum, or about a <NUM> wavelength in the color spectrum to about a <NUM> wavelength in the color spectrum, or the like. For example, the first color, second color, third color, and fourth color can be a blue color, a green color, a purple color, a yellow color, a red color, or an orange color, which are spread out substantially equally along the color spectrum.

In one example, referring to the first group of visual markers <NUM>, the first visual marker 241a. is a purple color, the second visual marker 241b is a blue color, the third visual marker 241c is a yellow color, and the fourth visual marker 241d is a green color. Referring to the second group of visual markers <NUM>, the first visual marker 242a is a yellow color, the second visual marker 242b is a green color, the third visual marker 241c is a blue color, and the fourth visual marker 241d is a purple color. Referring to the third group of visual markers <NUM>, the first visual marker 243a is a green color, the second visual marker 243b is a purple color, the third visual marker 243c is a blue color, and the fourth visual marker 243d is a yellow color. In this configuration, each of the four colors (blue, green, purple, and yellow) are evenly distributed and spaced from one another among the groups of visual markers. For example, a yellow visual marker is not directly adjacent to another yellow visual marker and a blue visual marker is not directly adjacent to another blue visual marker.

In this example described above, there are three purple visual markers, three blue visual markers, three yellow visual markers, and three green visual markers (i.e., markers with four different colors). While each of the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> is shown as including four visual markers, more generally, the attachment <NUM> can include any number of groups of visual markers including at least one visual marker. For example, the attachment <NUM> can include a first group of visual markers, a second group of visual markers, a third group of visual markers, and a fourth group of visual markers, with each group containing at least one visual marker. Further, the at least one visual marker in each group has a different color than the visual markers in the other groups. Further, while the attachment <NUM> (<FIG>) is shown as including twelve visual markers combined between the first group <NUM>, the second group <NUM>, and the third group <NUM>, it should be understood that the attachment <NUM> can include any number of visual markers (e.g., four visual markers, six visual markers, ten visual markers, twenty visual markers, fifty visual markers, etc.) having four or more different colors (e.g., four different colors, six different colors, ten different colors, twenty different colors, etc.). As will be discussed in more detail herein, having four or more visual markers having different colors is preferable to accurately and efficiently track motion of the tracking element <NUM>. Further, while the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> are each positioned along a circumferential length of the lower portion <NUM> and are evenly spaced from one another, the visual markers can be positioned relative to one another in any appropriate arrangement (e.g., randomly) on the outer surface of the lower portion <NUM>.

Referring to <FIG>, a method <NUM> for estimating a pose of the oral hygiene device <NUM> relative to a location includes, for example, a first step <NUM>, a second step <NUM>, a third step <NUM>, a fourth step <NUM>, a fifth step <NUM>, and a sixth step <NUM>.

The first step <NUM> includes receiving image data, from a camera that is the same as or similar to the camera <NUM> (<FIG>) described above, that is reproducible as an image of at least a portion of the oral hygiene device <NUM> and at least a portion of the tracking element <NUM>. For example, the image data can be a frame of a video image captured by the camera. As described above, the camera is positioned relative to the user (e.g., user <NUM>) such that the oral hygiene device <NUM> and the tracking element <NUM> are positioned between the camera and the user. Because the field of view of the camera encompasses the oral hygiene device <NUM>, the tracking element <NUM>, and at least a portion of the user, the captured video or still image includes at least a portion of all three and the background behind the user that is within the field of view of the camera.

The second step <NUM> includes analyzing the image data from the first step <NUM> to identify a region of interest within the image. Generally, the region of interest is an area of the image received during the first step <NUM> that includes the tracking element <NUM>. As described above, the image captured during the first step <NUM> includes at least a portion of the user and a background behind the user. By limiting the region of interest to an area surrounding the tracking element <NUM>, the processing requirements for the subsequent steps of the method <NUM> can be reduced.

To analyze the image data and identify the region of interest, one or more processors that are the same as or similar to the processor <NUM> (<FIG>) described above are used to identify the pattern <NUM> of the tracking element <NUM> (<FIG>) using a plurality of filters. The plurality of filters includes a movement filter, a color filter, and a shape filter. The movement filter detects or identifies movement within the image. Generally, the movement filter detects movement by distinguishing areas of movement in the image compared to stationary areas of the image. The movement filter takes advantage of the fact that the pattern <NUM> is likely to be moving due to corresponding movement of the oral hygiene device <NUM> and the tracking element <NUM> to narrow the potential area(s) of the image that could be the region of interest (i.e., contain at least a portion of the pattern <NUM>) by eliminating the stationary background of the image. The color filter and the shape filter identify the contrast in color between the background <NUM> and the plurality of indicators <NUM> and the shape of each of the plurality of indicators <NUM>. Having detected an area of the image containing the pattern <NUM>, the region of interest is defined as that area and excludes the remainder of the image.

Identifying the region of interest in a high resolution or high definition image requires substantial processing/computation time. To reduce the processing requirements for identifying the region of interest, the image analyzed during the second step <NUM> is preferably a low resolution image. The region of interest can then be upscaled to a higher resolution image for the remainder of the steps of the method <NUM>.

In some implementations, the pattern <NUM> of the tracking element <NUM> can be filtered or detected to identify the region of interest using a machine learning algorithm or an artificial intelligence algorithm. Machine learning algorithms may take a variety of forms. For example, the method <NUM> can utilize more basic machine learning tools such as a decision tree ("DT") or an artificial neural network ("ANN"). DT programs are generally used because of their simplicity and ease of understanding. DT are classification graphs that match input data to questions asked at each consecutive step in a decision tree. The DT program moves down the "branches" of the tree based on the answers to the questions. For example, a first branch may ask if a portion of the image is moving. If yes, a second branch may ask whether the portion of the image includes the pattern <NUM>. In other examples, deep learning algorithms or other more sophisticated machine learning algorithms can be used, such as, for example, a convolutional neural network.

Machine learning algorithms (e.g., a Haar Cascade) require training data to identify the features of interest that they are designed to detect. For instance, various methods may be utilized to form the machine learning models including applying randomly assigned initial weights for the network and applying gradient descent using back propagation for deep learning algorithms. In other examples, a neural network with one or two hidden layers can be used without training using this technique. In some examples, the machine learning algorithms will be trained using labeled data, or data that represents certain features, specific actions, or characteristics, including a particular color or a particular shape.

The third step <NUM> includes identifying candidate visual markers in the region of interest identified during the second step <NUM>. Generally, candidate visual markers are sub-regions of the region of interest that could be an actual visual marker (e.g., one of the visual markers of the groups of visual markers <NUM>, <NUM>, or <NUM> in <FIG>) on the tracking element <NUM>. To identify candidate visual markers, the region of interest is segmented in a plurality of sub-regions using a color segmentation algorithm. Each sub-region is defined by a plurality of pixels of the region of interest that have a common color.

Generally, the color segmenting algorithm assumes that objects are colored distinctively and seeks to identify gross color differences between adjacent pixels in an image. The color segmenting algorithm uses the L*a*b color space, which defines colors in terms of luminosity ("L"), where the color falls along the red-green axis ("*a"), and where the color falls along the blue-yellow axis ("*b"). As a result, if necessary, the region of interest identified in the second step <NUM> is converted from a RGB color space to the L*a*b color space to perform the color segmenting algorithm. Using a threshold value, the color segmenting algorithm separates adjacent pixels having distinctive colors from one another to form a plurality of sub-regions. The average color in the L*a*b color space of each of the plurality of sub-regions is then calculated.

As discussed above, the color of the visual markers in the first group <NUM>, the second group <NUM>, and the third group <NUM> preferably is one of blue, green, purple, yellow color, red, or orange. Thus, a sub-region of the region of interest having a blue, green, purple, or yellow color could be a candidate visual marker. While the region of interest is limited to an area encompassing the tracking element <NUM>, the region of interest may still include a portion of the user or the background behind the user, which can create false positive for a candidate visual marker. For example, the user may be wearing clothing which has one or more of the same or similar colors as the visual markers.

To increase the accuracy of identifying candidate visual markers, the third step <NUM> also includes a shape filter and a size filter. The visual markers of the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> (<FIG>) have a generally dome-like or hemispheric shape. When viewed in a two-dimensional image such as the region of interest, these visual markers have a generally circular shape. The shape filter and the size filter are used to detect the generally circular shape of the visual markers within the region of interest. These filters aid in discriminating between a visual marker and, for example, the clothing of the user.

The fourth step <NUM> includes obtaining a first proposed three-dimensional pose of the oral hygiene device <NUM>. Generally, the first proposed three-dimensional pose includes the position and orientation (rotation and translation) of the oral hygiene device <NUM> relative to the camera. As will be discussed in more detail herein, in some implementations, the fourth step <NUM> will not initialize until at least four candidate visual markers are identified during the third step <NUM>. If less than four candidate visual markers are identified during the third step <NUM>, the method <NUM> is repeated until at least four candidate visual markers are identified.

Referring to <FIG>, the fourth step <NUM> includes a first sub-step <NUM>, a second sub-step <NUM>, a third sub-step <NUM>, and a fourth sub-step <NUM>.

The first sub-step <NUM> includes grouping the candidate visual markers identified during the third step <NUM> (<FIG>) into discrete sets of candidate visual markers. Preferably, each of the discrete sets of candidate visual markers includes four candidate visual markers.

Similarly, the second sub-step <NUM> includes grouping model markers from a three-dimensional model of the oral hygiene device <NUM> and the tracking element <NUM> into discrete steps. The three-dimensional model is stored in a memory device that is the same as or similar to the memory device <NUM> described above (<FIG>). The three-dimensional model is a representation of the actual oral hygiene device <NUM> and the tracking element <NUM>. Specifically, the three-dimensional model includes representations of the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM>. The number of model markers in each of the discrete sets of model markers is equal to the number of candidate visual markers in the first discrete set of candidate visual markers grouped together during the first step <NUM> (e.g., four candidate visual markers and four model markers).

The third sub-step <NUM> includes selecting a first discrete set of candidate visual markers and a first discrete set of model markers. The first discrete set of candidate visual markers includes four visual markers and the first discrete set of model markers includes four model markers.

The fourth sub-step <NUM> includes inputting the first discrete set of candidate visual markers and the first discrete set of model markers selected during the third sub-step <NUM> into a perspective-three-point ("P3P") algorithm. The P3P algorithm is based on the law of cosines and is used to estimate an object pose (a rotation and translation) relative to the camera placement.

Generally, the P3P algorithm compares two-dimensional points taken from an image with three-dimensional points taken from a three-dimensional model. To solve the P3P equation system, four two-dimensional points defined in an image coordinate system and four three-dimensional points defined in a three-dimensional model coordinate system are provided. Three sets of points, each set including a two-dimensional point and a three-dimensional point, are used to solve the P3P equation system and determine up to four possible sets of distances between the two-dimensional points and the optical center of the camera. These four sets of distances are converted into four pose configurations. The fourth set of 2D/3D points is then used to select the best or most correct pose configuration against the four proposals. There are various methods for solving the P3P equation system and obtaining an estimated three-dimensional pose. For example, one such method is explained in <NPL>).

Inputting the first discrete set of candidate visual markers and the first discrete set of model markers into the P3P algorithm and solving the equation system yields a first proposed three-dimensional pose of the oral hygiene device <NUM>. The first proposed three-dimensional pose includes a rotational and a translational position of the oral hygiene device <NUM> that permits the position of the oral hygiene device <NUM> relative to the camera to be determined. As discussed above, in some implementations, the fourth step <NUM> will not initialize until four candidate visual markers are identified during the third step <NUM>. This is because solving the P3P algorithm requires four candidate visual markers and four model markers. If less than four candidate visual markers are identified in the third step <NUM>, the P3P algorithm equation system generally cannot be solved without more data.

The fifth step <NUM> (<FIG>) includes validating the first proposed three-dimensional pose of the oral hygiene device <NUM> determined during the fourth step <NUM>. It is possible that the first discrete set of candidate visual markers and the first discrete set of model markers selected in sub-step <NUM> yield a proposed three-dimensional pose that is incorrect (e.g., a pose that is not physically possible). Thus, the fifth step <NUM> is generally used to validate or reject the proposed three-dimensional pose obtained during the fourth step <NUM>.

Referring to <FIG>, the fifth step <NUM> includes a first sub-step <NUM>, a second sub-step <NUM>, a third sub-step <NUM>, and a fourth sub-step <NUM>. The first sub-step <NUM> includes predicting the positions of the candidate visual markers within the region of interest. Based on the first proposed three-dimensional pose calculated during the fourth step <NUM>, and the known positions of visual markers from the three-dimensional model of the oral hygiene device <NUM> and the tracking element <NUM>, the position of the visual markers within the region of interest can be predicted. In other words, the predicted positions indicate where candidate visual markers should be located in the region of interest if the first proposed three-dimensional pose is correct, and where candidate visual markers should not be located. For example, it may be predicted that six visual markers will be visible in the region of interest if the oral hygiene device <NUM> has the same pose as the first proposed three-dimensional pose. The position of these six visual markers relative to one another in the region of interest is determined from the three-dimensional model of the oral hygiene device.

The second sub-step <NUM> includes comparing the candidate visual markers identified in the region of interest with the predicted positions of the visual markers. More specifically, the number and position of the candidate visual markers is compared to the predicted number and predicted position of the visual markers (first sub-step <NUM> of the fifth step <NUM>). If it is determined that the positions of a predetermined number of the candidate visual markers correspond to the predicted positions, the first proposed three-dimensional pose is validated (sub-step <NUM>). If less than the predetermined number of candidate markers correspond to the predicted positions, the first proposed three-dimensional pose is rejected (sub-step <NUM>).

To illustrate by way of an example, the first sub-step <NUM> predicts that six candidate visual markers will be visible in the region of interest, and predicts the position of each of these six candidate visual markers relative to one another. The third step <NUM> identified ten candidate visual markers within the region of interest. If, for example, a candidate visual marker corresponds to five of the six predicted visual markers, the first proposed three-dimensional pose is validated (sub-step <NUM>) and the fifth step <NUM> is completed. The other four candidate visual markers are simply considered to be noise or inaccurate. Alternatively, if there are thirty candidate visual markers identified during the third step <NUM>, and for example, twenty-five of the thirty do not correspond to a predicted position, the proposed three-dimensional pose may be rejected.

The predetermined number of correspondences required to validate a proposed three-dimensional pose can be expressed as a percentage, and can be at least about <NUM>% of the predicted positions of the visual markers correspond to positions of candidate visual markers, at least about <NUM>% of the predicted positions of the visual markers correspond to positions of candidate visual markers, at least about <NUM>% of the predicted positions of the visual markers correspond to position of candidate visual markers, at least about <NUM>% of the predicted positions of the visual markers correspond to positions of candidate visual markers, at least about <NUM>% of the predicted positions of the visual markers correspond to positions of candidate visual markers, or <NUM>% of the predicted positions of the visual markers correspond to positions of candidate visual markers). In some implementations, the predetermined number is a statistically significant number such that it can be determined that the first proposed three-dimensional pose is correct with a suitable statistical certainty (e.g., <NUM>% statistical certainty, <NUM>% statistical certainty, <NUM>% statistical certainty, etc.).

If the first proposed-three dimensional pose is rejected (sub-step <NUM>), the second sub-step <NUM> (<FIG>) of the fourth step <NUM> is repeated. During the repeating of the sub-step <NUM>, a second discrete set of candidate visual markers and a second discrete set of model markers are selected. At least one of the second discrete set of candidate visual markers and the second discrete set of models markers includes a set of candidate visual markers or model markers that is different from the first discrete set of candidate visual markers and/or the second discrete set of model markers. These sets are then inputted into the P3P algorithm in sub-step <NUM> to obtain a second proposed three-dimensional pose of the oral hygiene device <NUM>. The second proposed three-dimensional pose is then validated or rejected during the fifth step <NUM>. Steps <NUM> through <NUM> are repeated until a proposed three-dimensional pose is validated (sub-step <NUM>).

During the repeating of the steps described above to validate a proposed three-dimensional pose, numerous discrete sets of candidate visual markers and discrete sets of model markers may be inputted into the P3P algorithm until a proposed pose is validated. Because there are twelve visual markers collectively between the first group of visual markers <NUM> in the example shown in <FIG>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> (<FIG>), there are <NUM> possible combinations of four visual markers if the color of the visual markers is disregarded. In other words, there are <NUM> possible discrete sets of four model markers. If for example, there are sixteen candidate visual markers identified in the region of interest (during step <NUM>), disregarding color, there are <NUM>,<NUM> combinations of four candidate visual markers (i.e., <NUM>,<NUM> possible discrete sets of candidate visual markers). This means that there is potentially over <NUM>,<NUM> proposed three-dimensional poses that may be determined before one is validated, requiring substantial processing/computation time. However, as described above, each of the first group of visual markers <NUM>, the second group of visual markers <NUM>, and the third group of visual markers <NUM> includes four visual markers, and each of the four visual markers in each group has a different color. The grouping of candidate visual markers and model markers can then be further conditioned such that each group not only is limited to four visual markers, but each visual marker in the group of four has a different color. In this manner, the number of possible combinations that may need to be inputted into the P3P algorithm (sub-step <NUM>) before validating a proposed three-dimensional pose (sub-step <NUM>) is reduced from, for example, the hundreds of thousands to several hundred. This reduces the processing/computational requirements such that the method can be implemented on, for example, a smartphone with limited processing power.

The sixth step <NUM> includes obtaining a second proposed three-dimensional pose of the oral hygiene device <NUM> based on the validated first proposed three-dimensional pose of the oral hygiene device <NUM>. The second proposed three-dimensional pose of the oral hygiene device <NUM> is calculated in a similar manner as the first proposed three-dimensional pose during the fifth step <NUM>. As discussed above, during the first sub-step <NUM> of the fifth step <NUM>, the positions of each of the visual markers in the region of interest are predicted. As also discussed above, there may be a greater number of candidate visual markers identified during the third step <NUM> than the amount of predicted visual markers due to noise from the background or inaccuracy involved in the color segmenting algorithm. To obtain a more refined pose estimation, the sixth step <NUM> selects only the candidate visual markers ("correct candidate visual markers") that correspond to predicted visual markers, ignoring candidate visual markers that are incorrect based on the predicted positions. These correct candidate visual markers are then compared to model markers from the three-dimensional model of the oral hygiene device <NUM> using an iterative pose estimation algorithm and linear regressions to obtain a second proposed three-dimensional pose of the oral hygiene device <NUM>. The second proposed three-dimensional pose of the oral hygiene device <NUM> is generally more accurate than the first proposed three dimensional pose (fourth step <NUM>), but requires more processing/computation time to determine.

Referring to <FIG>, after completion of the sixth step <NUM>, the method <NUM> can be repeated one or more times. In a second iteration of the method <NUM>, the first step <NUM> is repeated and includes receiving image data that is reproducible as a second image of at least a portion of the oral hygiene device <NUM> and the tracking element <NUM>. For example, the second image can be a second frame of a video image that is taken subsequent to the image used during the initial iteration of the method <NUM>.

The second step <NUM> is then repeated to identify a second region of interest in the second image received during the first step <NUM>. However, in the second iteration of the method <NUM>, detection of the pattern <NUM> of the tracking element <NUM> to identify the region of interest is bypassed. Instead the second region of interest is selected using the second three-dimensional pose estimation (sixth step <NUM>), and the second region of interest is defined an area of the second image in which at least a portion of the tracking element <NUM> is positioned. Because the second step <NUM> in the second iteration of the method <NUM> does not require detection of the pattern <NUM> using a plurality of filters, the required processing/computation time to complete the second step <NUM> is reduced.

The third step <NUM> is then repeated to identify all of the candidate visual markers in the second region of interest using the color segmenting algorithm described above. Typically, the oral hygiene device <NUM> will be used in a bathroom that may have bright or intense lighting, which can be further amplified by reflections in a bathroom mirror. Further, movement of the oral hygiene device <NUM> may cause the lighting conditions in the region of interest to change based on position of the oral hygiene device relative to a light source (e.g., the user may cast a shadow on a portion of the oral hygiene device <NUM> in a particular pose). The lighting conditions and/or movement of the oral hygiene device <NUM> may affect the amount of light reflecting off of the visual markers of the tracking element <NUM>. For example, it may be difficult to discern a blue color from a purple under intense or bright lighting conditions or dark lighting conditions. By using the second three-dimensional pose estimation obtained during the sixth step <NUM>, the threshold for distinguishing colors in the color segmenting algorithm can be adjusted based on the three-dimensional pose estimation obtained in the sixth step <NUM> of the first iteration of the method <NUM>. This threshold is then updated each time the third step <NUM> is completed as the method <NUM> is repeated.

The fourth step <NUM>, the fifth step <NUM>, and the sixth step <NUM> are then repeated in the same or similar manner as described above to obtain another second three-dimensional pose estimation of the oral hygiene device <NUM>.

Steps <NUM> through <NUM> can then be repeated a plurality of times (e.g., ten times, fifty times, one hundred times, one thousand times, etc.) after the second iteration described above description to track motion of the oral hygiene device <NUM>. The sixth step <NUM> will output a series of estimated three-dimensional poses of the oral hygiene device <NUM> as the method <NUM> is repeated, which can be then used to track the movement of the oral hygiene device <NUM> over time. This repeating of the method <NUM> can be used to track the motion of the oral hygiene device <NUM> during, for example, a brushing session in which a user is brushing their teeth. Data relevant to the quality of brushing by a user or the overall dental health of the user's teeth can be collected and analyzed based on the motion data. For example, a brush stroke type (e.g., a side-to-side stroke, an angular stroke, or a circular stroke) can be determined.

In some implementations, the system <NUM> can also be utilized to determine the position and orientation of a face of a user. For example, using the camera <NUM>, the system <NUM> receives an image of at least a portion of the face of the user. Using the processor <NUM> and the memory <NUM> and a facial recognition algorithm, the position and orientation of the face can be determined. For example, the system <NUM> may determine the position of the user's eyes, mouth, or nose (or any combination thereof) using, for example, a plurality of filters, machine-learning algorithms, or the like. In one example, the position of the user's mouth can be estimated based on the position of the user's eyes and a distance between the eyes and mouth of the user.

By determining the position and orientation of the face of the user, the position of the oral hygiene device <NUM> can be determined not only with respect to the camera, but to the mouth of the user. Thus, the position of the oral hygiene device relative to the teeth of the user and thus determine and whether a user has a brushed a certain section of teeth can be determined.

In some implementations, the method <NUM> further includes an initial calibration step to determine the rotational position of the tracking element <NUM> relative to the oral hygiene device <NUM>. For example, using the techniques described above, the calibration step can initially determine the rotational position of the tracking element <NUM> on the oral hygiene device <NUM> and communicate that position to adjust the three-dimensional model so that the rotational position of the tracking element <NUM> in the three-dimensional model corresponds with the rotational position on the actual oral hygiene device <NUM>. In other implementations, the method <NUM> is agnostic to how the tracking element <NUM> is coupled to the handle <NUM> of the oral hygiene device <NUM>.

Advantageously, the tracking element <NUM> can be used to track motion of the oral hygiene device <NUM> using the method <NUM> (or other similar methods) without requiring any electronics or sensors (e.g., an accelerometer) in the tracking element <NUM>. While the tracking element <NUM> can include such sensors in some implementations to aid in tracking motion of the oral hygiene device <NUM>, such sensors may, for example, increase the cost of the tracking element <NUM>, require the tracking element <NUM> to be charged periodically prior to use, or increase the weight of the tracking element <NUM> and thus interfere with a user's (e.g., a child's) brushing given the added weight at the end of the oral hygiene device <NUM>.

While the system <NUM> and method <NUM> have been illustrated and described herein as being used to track the motion of an oral hygiene device (e.g., oral hygiene device <NUM>), the system <NUM> and method <NUM> can be used to track the motion of any other object coupled to the tracking element <NUM>. For example, a tracking element that is the same as or similar to the tracking element <NUM> can be coupled to an end of an object with a similar shape as the oral hygiene device <NUM>, such as, for example, a baseball bat, a hockey stick, a golf club, or the like. Further, a tracking element that is similar to the tracking element <NUM> can more generally be attached to an object with any other shape to track the motion of the object.

Claim 1:
A method for estimating a pose of an oral hygiene device (<NUM>; <NUM>) relative to a location, the oral hygiene device (<NUM>; <NUM>) including a pattern (<NUM>, <NUM>) and a plurality of groups of visual markers (<NUM>, <NUM>-<NUM>), the method comprising:
a) receiving image data reproducible as an image of at least a portion of the oral hygiene device (<NUM>; <NUM>);
b) analyzing, using one or more processors (<NUM>), the image data to identify a region of interest within the image, the region of interest including at least a portion of the pattern (<NUM>, <NUM>) therein;
c) responsive to identifying the region of interest, segmenting, using at least one of the one or more processors (<NUM>), the region of interest into a plurality of sub-regions, each of the plurality of sub-regions being defined by a plurality of pixels having a common color;
d) identifying, using at least one of the one or more processors (<NUM>), all candidate visual markers from the plurality of groups of visual markers (<NUM>, <NUM>-<NUM>) within the region of interest, wherein candidate visual markers correspond to the sub-regions of the region of interest;
e) creating a plurality of distinct sets of the candidate visual markers;
f) selecting a first one of the plurality of distinct sets of the candidate visual markers;
g) selecting a first one of a plurality of distinct sets of model markers associated with a three-dimensional model of the oral hygiene device (<NUM>, <NUM>);
h) evaluating the selected set of the candidate visual markers and the selected set of model markers using a perspective-three-point algorithm to obtain a proposed three-dimensional pose of the oral hygiene device (<NUM>, <NUM>);
i) based on the proposed three-dimensional pose of the oral hygiene device (<NUM>, <NUM>), predicting a position within the region of interest for a predetermined number of the candidate visual markers;
j) comparing the predicted positions for the predetermined number of the candidate visual markers with actual positions of all of the candidate visual markers within the region of interest;
k) responsive to a determination that at least a substantial portion of the predicted positions correspond with the actual positions, validating the proposed three-dimensional pose; and
l) responsive to a determination that less than the substantial portion of the predicted positions correspond with the actual positions, repeating steps f) - k).