Patent Description:
<NPL> discloses TMotion, a self-contained 3D input that enables spatial interactions around mobile device using a magnetic sensing technique. A permanent magnet and an inertial measurement unit (IMU) is embedded in a stylus. When the stylus moves around the mobile device, a continuous magnetometer readings is obtained. By numerically solving non-linear magnetic field equations with known orientation from IMU, 3D position tracking with update rate greater than <NUM> is achieved. Experiments evaluated the position tracking accuracy, showing an average error of <NUM> in the space of <NUM> x <NUM> x <NUM>. Furthermore, the experiments confirmed the tracking robustness against orientations and dynamic tracings. In task evaluations, the tracking and targeting performance in spatial interactions are verified with users. Example applications that highlight TMotion's interaction capability are demonstrated.

<CIT> discloses various embodiments which include systems and methods for rendering images in a virtual or augmented reality system that may include capturing scene images of a scene in a vicinity of a first and a second projector, capturing spatial data with a sensor array in the vicinity of the first and second projectors, analyzing captured scene images to recognize body parts, and projecting images from each of the first and the second projectors with a shape and orientation determined based on the recognized body parts. Additional rendering operations may include tracking movements of the recognized body parts, applying a detection algorithm to the tracked movements to detect a predetermined gesture, applying a command corresponding to the detected predetermined gesture, and updating the projected images in response to the applied command.

This summary is not intended to exclusively identify key features or essential features of the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.

Systems and methods for accurate tap event location within a scene in a device view frustum that is modeled by a <NUM>-D map include embodiments that comprise a device for use with a selection apparatus. The device and the selection apparatus may be used to select a subject within a scene in the view frustum of the device, by tap event detection and location. Determination of the location of the selection apparatus during the tap event, as the selection apparatus contacts a target subject, allows an accurate determination of the subject that is selected. In various implementations of the system and method the determination of the subject may be used in further processing by applications on the device.

In an embodiment, the device determines a <NUM>-dimensional (<NUM>-D) map of a scene in the view frustum of the device relative to a coordinate frame based on inputs from at least one sensor. The device receives an indication of the occurrence of a tap event from the selection apparatus, and determines the location of the tap event relative to the coordinate frame by determining the location of the selection apparatus relative to the device at the time of the tap event. The device then translates the location of the tap event into the <NUM>-D map coordinate frame. The device may then determine the subject of the tap event based on the location of the tap event in the coordinate frame. The subject may comprise, for example, a point, a surface, an object, or a virtual object shown within the view frustum of the device. The selection apparatus may be configured to communicate occurrences of tap events to the device and include features that allow the device to determine the location of the tap event by determining the location of the selection apparatus. The determination of the location of the selection apparatus may include determination of the location of a particular point on, or portion of, the selection apparatus at which the tap event takes place.

The location of the tap event may be used to determine an accurate point location, an object, a surface, or a portion of a surface in the <NUM>-D map. The use of the selection apparatus to perform a tap on or near an object or surface that triggers an indication of the tap event to the device, and, the ability of the device to locate the selection apparatus at the point of contact provides an accurate location of the tap event. Location data associated with the point location, the object, the surface, or the portion of a surface may then be processed further to provide effects in, or data about, the scene in the view frustum. For example, the location data may be used to implement application features in an augmented reality device that receives selections of objects, surfaces, or points in the device view frustum. The selected objects, surfaces, or points may then be manipulated in the augmented reality created by the device.

Example embodiments also include a selection apparatus including a tap event sensor that detects a tap event comprising a contact of the selection apparatus with an object or surface. The selection apparatus may include a module for communicating an indication of the occurrence of the tap event to another device remote to the object or surface. The selection apparatus may also include a fiducial marker to allow the other device to determine a location of the selection apparatus at the time of the tap event when the selection apparatus is located within a view frustum of the other device. In an implementation, the selection apparatus may be configured as a stylus including a tip, and the tap event sensor may be coupled to the tip of the stylus. The fiducial marker may be disposed at the tip of the stylus to allow the other device to determine an accurate location of the point of the tap event.

The system, method and apparatus will now be described by use of example embodiments. The example embodiments are presented in this disclosure for illustrative purposes, and not intended to be restrictive or limiting on the scope of the disclosure or the claims presented herein.

As used herein, a tap event occurs when a user taps a real world object with the tip of a selection apparatus (such as a stylus). A tap event may require that the stylus be placed in a tap-event mode or it may occur whenever the tip of the stylus detects that it has contacted a real world object (such as by using a pressure sensor on the tip of the stylus or by measuring the movement of the stylus using accelerometers to detect a sudden change). The stylus can then store the real world coordinates of the tap event (this is referred to as the tap even location) in memory.

The tap event location may provide advantages when used for subject selection in a real world scene within the view frustum of a device. In an implementation, a device may store a <NUM>-D map that models the real world scene relative to a reference coordinate frame. The embodiments allow the user to obtain accurate location data through the device detecting and physically locating tap events, relative to the reference coordinate frame of the <NUM>-D map. The tap events may be performed by tapping selected subjects such as points, objects or surfaces in the scene with a selection apparatus. The selection apparatus may be configured to provide an indication of the occurrence of the tap event to the device which allows the device to accurately determine the location at which the tap event occurred. The selection apparatus may be configured with features, such as a unique fiduciary marker, that enhance the ability of the device to accurately distinguish and locate the position of the selection apparatus within a <NUM>-D scene that includes real world and simulated objects upon detecting that a tap event has occurred.

The location data of tap events in the coordinate frame may be used to accurately select subjects represented in the <NUM>-D model for further processing or manipulation by device applications. Based on the location of the tap event, the device may determine, from a device use context defined by a device application, that an object or structure that includes a surface represented by the <NUM>-D mesh was selected as the subject. The device may also determine from the device use context that only a portion of the surface represented by the <NUM>-D mesh, and upon which the tap event occurred, was selected as the subject. The selected subject may also be a virtual object, such as an object projected into the scene by an augmented reality (AR) device.

Accurately locating the position of a tap event may provide advantages to a device that allows a user to perform input in a real world scene modeled by the device. For example, the positions of tap events may be used in an augmented reality (AR) device in which a composite image, with mixed real and non-real objects, is used to present a scene to a viewer. When used with an AR device, the tap event detection and location method of the embodiments allows more accurate selection of points, objects or surfaces as compared to selection methods that do not use the selection apparatus of the embodiments. The selection apparatus allows a user to provide a clear signal to an AR device that selection of a point, object or surface is intended by providing the AR device an accurate time and accurate location of the tap event. For example, a tap event may provide accurate differentiation between certain target objects or surfaces that are close to other objects or surfaces nearby in the real world. In one configuration, a user may desire to select a target object that is small compared to a background object on which it is placed. In this case, the user may tap the target object with the selection apparatus (this tap is the tap event) which then sends an indication of the tap occurrence to the AR device at the time of the tap event. The AR device may then locate the selection apparatus at the time of the tap event, determine the location of the tap event, and determine that the target object is to be selected. The AR device has a clear indication that selection of the object, i.e., target object, at the tap event location is intended.

In an example implementation using tap event location in a device and selection apparatus, the device may adjust and/or refine already stored coordinate location data defining a <NUM>-dimensional mesh that models a physical surface in its <NUM>-D map based on more accurate location data determined from tap events on the actual physical surface. This may be performed by determining the locations of tap events on the actual surface modeled by the <NUM>-D mesh in the coordinate frame and comparing those tap event locations with location coordinates of the <NUM>-D mesh in the <NUM>-D model. The coordinate locations of the <NUM>-D mesh stored in the <NUM>-D model may then be adjusted to remove any inconsistencies between the tap event locations performed on the physical surface and the location coordinates of the <NUM>-D mesh modeling the surface. This allows a user to fine tune the <NUM>-D model of a real world scene as desired.

In further example implementations, location data of tap events may also be used to accurately perform measurements between points in a scene modeled by a <NUM>-D map. The distance between two tap events performed at physical points in the scene may be used to determine and accurately measure the physical distance between the two points. For example, the <NUM>-D map stored in a device may comprise location data associated with a first and a second <NUM>-D mesh that represent, respectively, a first and a second surface in the device view frustum. The device may determine the occurrence of successive first and second tap events, performed on the first and second surfaces, respectively. The device may determine that the subject of the first tap event was the first surface based on the location of the first tap event relative to the first <NUM>-D mesh location and determine that the subject of the second tap event was the second surface based on the location of the second tap event relative to the second <NUM>-D mesh location. The device may then determine a distance between the first and second surface based on the <NUM>-D map. For example, depending on the device use context, or a currently activated application, the device may determine the shortest or the longest distance between the first and second surfaces. Measurements made according to this implementation may be performed using any type of surfaces such as walls or curved surfaces. The device may also determine the shortest or longest distance between portions of the two surfaces. This implementation may be used, for example, when measuring between two walls since it doesn't require the first and second tap events to be directly opposite one another on the two walls for an accurate distance measurement.

Referring now to <FIG>, therein is illustrated an example implementation of a device <NUM> and a selection apparatus <NUM>. <FIG> illustrates an example scenario in which a user <NUM> is utilizing the device <NUM> in conjunction with the selection apparatus <NUM>. In <FIG>, device <NUM> is implemented as a wearable headset device configured to provide augmented reality (AR) functions. Selection apparatus <NUM> may be implemented as a stylus type device that is configured to detect a tap event and generate an indication of the tap event for device <NUM>. In an example implementation, selection apparatus <NUM> may include a tip <NUM> that is coupled to a tap sensor internal to selection apparatus <NUM>. Selection apparatus <NUM> may also include a fiducial marker that is disposed on a casing of selection apparatus <NUM> near tip <NUM>. <FIG> illustrates an example view frustum provided to user <NUM> by device <NUM>. The view frustum is defined by sight lines e, f, g and h that radiate out at angles from device <NUM> depending on the direction in which user <NUM> points device <NUM>. The view frustum of device <NUM> may extend out along sight lines e, f, g and h as far out from device <NUM> as sensors on device <NUM> are able to detect objects in a scene. Lines a, b, c and d illustrate an example area of the view frustum at a particular distance from device <NUM>. The view frustum extends to distances beyond the example area outlined by lines a, b, c and d. The view frustum provides a real world scene of objects to user <NUM>. Device <NUM> determines and stores a <NUM>-D map of the scene relative to a reference coordinate frame <NUM> represented by example axes <NUM>, <NUM>, and <NUM>. Device <NUM> also determines its own location and orientation with respect to the reference coordinate frame. User <NUM> may hold selection apparatus <NUM> in their hand and select objects or surfaces within the view frustum, i.e., the area outlined by lines a, b, c and d, of device <NUM> by tapping the objects with selection apparatus <NUM>. In alternative implementations of <FIG>, device <NUM> may be configured as any type of mobile computing device, a smart phone, a tablet device, a static computing device, a system of components such as studio equipment, or any other device or system of components that interacts with a selection apparatus according to the embodiments.

<FIG> is a simplified block diagram illustrating portions of the device <NUM> and selection apparatus <NUM> of <FIG>. Device <NUM> includes communication module <NUM>, tap event locator <NUM>, sensors <NUM>, <NUM>-D mapper <NUM>, tap event subject determiner <NUM>, and application programs <NUM>. Selection apparatus <NUM> may be configured as a stylus type device including casing 102a having a tip <NUM>, and fiducial marker <NUM>. Block 102b shows example components of device <NUM> which may be configured on and/or within casing 102a, including the tip <NUM>, tap sensor <NUM> and communication module <NUM>. Fiducial marker <NUM> may comprise a specially configured retroreflector or colored LED to allow a device, such as device <NUM>, to visually distinguish the location of the tip of the stylus within the scene in the view frustum of device <NUM>. Fiducial marker <NUM> may be disposed at or near the tip <NUM> of selection apparatus <NUM>, where a tap occurs, to allow another device, such as device <NUM>, to determine an accurate location of the tap event. Tap sensor <NUM> may be coupled to the tip <NUM> and be activated when a tap event occurs.

Referring now to <FIG>, therein is a flow diagram illustrating example operations performed by a device according to an embodiment of the disclosure. The device of <FIG> may be implemented as device <NUM> of <FIG> and the operations of <FIG> may be described with reference to <FIG>.

The process of <FIG> begins at <NUM> where device <NUM> determines a <NUM>-D map of a real world scene within the view frustum of the device <NUM>. The view frustum may comprise any scenes or surroundings within the range of sensors <NUM> and in the area outlined by view lines e, f, g and h as the view lines extend outward from device <NUM>. The scene may be changing and be updated as user <NUM> changes the position of device <NUM>.

The sensors <NUM> of device <NUM> may be used to sense and receive data associated with a scene within the view frustum <NUM> and provide the data <NUM> as sensed data <NUM> to <NUM>-D mapper <NUM>. The sensors <NUM> may comprise, for example, a depth camera, a red, green, blue (RGB) camera, and an inertial measurement unit (IMU). The <NUM>-D mapper <NUM> may utilize the sensed data <NUM> to map the real world scene onto a <NUM>-D map relative to the reference coordinate frame <NUM>. <NUM>-D mapper <NUM> may also determine the location and orientation of device <NUM> relative to the reference coordinate frame <NUM> by self-tracking device <NUM> as user <NUM> moves.

The <NUM>-D map may comprise location data associated with objects and surfaces in the real world scene sensed by sensors <NUM> relative to the reference coordinate frame <NUM>. The <NUM>-D map may also comprise data associated with <NUM>-D meshes that model one or more surfaces of the real world scene. In example implementations, the <NUM>-D map may also comprise data associated with virtual objects that are inserted into the real world scene or data associated with virtual manipulations of real world objects in the real world scene. For example, an AR device may augment the real world scene for a device user by inserting virtual objects into the scene, or changing characteristics of real world objects to appear different in the scene.

At <NUM>, device <NUM> receives an indication from selection apparatus <NUM> that a tap event has occurred in the device view frustum. The indication is received by device <NUM> at communication module <NUM> that is configured to receive wireless communications <NUM> from communication module <NUM> of selection apparatus <NUM>. Communication module <NUM> may then provide an indication of a tap event <NUM> to tap event locator <NUM>.

At <NUM>, in response to receiving the indication of a tap event <NUM>, tap event locator <NUM> determines the location of the tap event in the reference coordinate frame <NUM>. Tap event locator <NUM> may determine the location of the tap event by utilizing data <NUM> provided by sensors <NUM> to locate selection apparatus <NUM> at the time of the tap event relative to the location of device <NUM>. Tap event locator <NUM> may use the known location and orientation of device <NUM> in reference coordinate frame <NUM> provided by <NUM>-D mapper/tracker <NUM> to translate the tap event's location relative to device <NUM> into the reference coordinate frame <NUM>. The determination of the location of the device <NUM> by tap event locator <NUM> may occur immediately upon receiving the indication of the tap event <NUM> in order that the location of the tap event is accurately determined. In an implementation, tap event locator <NUM> and sensors <NUM> may be configured to detect fiducial marker <NUM> on selection apparatus <NUM> to determine the tap event location. Fiducial marker <NUM> may be disposed on selection apparatus <NUM> adjacent to the tip <NUM> which contacts the targeted object or surface during the tap event. In an alternative implementation to <FIG>, the selection apparatus <NUM> may be self-locating and signal its location to device <NUM> along with the indication that a tap event has occurred without device <NUM> needing to determine the location of the selection apparatus <NUM>.

At <NUM>, tap event subject determiner <NUM> determines the subject of the tap event based on the location of the tap event <NUM> provided by tap event locator <NUM> and data <NUM> provided by <NUM>-D mapper <NUM>. The subject selected by the tap event may depend on device use context and on a particular application program of application programs <NUM> that is processing the tap event location data.

In the embodiments, the tap event location determined by tap event locator <NUM> at <NUM> allows a particular application or program having a particular use context to more efficiently determine the subject of the tap event. For example, the subject selected by the tap event may be determined as a physical point, a surface or portion of a surface, an object or portion of an object, or a virtual object. In other implementations, depending on the application, multiple subjects, such as like appearing objects, may be selected based on one tap event that selects of the multiple subjects.

At <NUM>, an application program <NUM> performs further processing in interaction with the other functions of device <NUM> using the determined subject of the tap event from operation <NUM>. The processing may be any processing that uses, analyzes, or manipulates the selected subject in the view frustum scene modeled by the <NUM>-D map in the device <NUM> to achieve an intended result.

Referring now to <FIG>, therein is a flow diagram illustrating example operations performed by a selection apparatus according to an embodiment of the disclosure. The selection apparatus of <FIG> may be implemented as selection apparatus <NUM> of <FIG> and the operations of <FIG> may be described with reference to <FIG>. Selection apparatus <NUM> may interact with device <NUM> as device <NUM> performs the operations of <FIG>. As illustrated in <FIG>, selection apparatus <NUM> may be configured as a stylus type device comprising case 102a including tip <NUM> to serve as the point of contact when an object or surface is tapped.

The process of <FIG> begins at <NUM> when an object or surface is tapped with tip <NUM> of selection apparatus <NUM>. At <NUM>, tap sensor <NUM> receives an indication <NUM> generated by the tap event at tip <NUM> and detects that the tap event has occurred. Tap sensor <NUM> may be implemented as a switch or pressure sensor that is activated by contact of tip <NUM> with the tapped object. Alternately, tap sensor <NUM> may be implemented as a microphone/acoustic sensor that senses the sound made by the tap of tip <NUM> on the selected object. In other alternative implementations, tap sensor <NUM> may include an accelerometer that is configured to sense contact of tip <NUM> with an object by sensing cessation of movement of selection apparatus <NUM> when selection apparatus <NUM> contacts the object. Also, an implementation of tap sensor <NUM> including an accelerometer may be used to sense tap selection of a virtual object. For example, the tap selection of a virtual object may be performed by sensing cessation of quick movement of selection apparatus <NUM> by a user at a point on a virtual object displayed to user <NUM> in the view frustum of device <NUM>.

At <NUM>, in response to determining that a tap event has occurred, the selection apparatus <NUM> generates an indication of the tap event to device <NUM>. Generation of the indication of the tap event to device <NUM> includes communication module <NUM> receiving an indication of a tap event <NUM> from tap sensor <NUM> and sending the indication as a wireless signal <NUM> from communication module <NUM> to communication module <NUM> of device <NUM>. This may be the indication that a tap event has occurred that is received by device <NUM> at operation <NUM> of <FIG>.

Referring now to <FIG>, therein is illustrated an example scene in the view frustum of a device implemented using a selection apparatus according to <FIG>. <FIG> illustrates a scene, for example, within the view frustum of device <NUM> of <FIG>. In the example implementation of <FIG>, device <NUM> determines <NUM>-D map of scene <NUM> relative to a reference coordinate frame represented by the axis shown as lines <NUM>, <NUM> and <NUM>. The scene <NUM> includes a view of a room with a desk <NUM>, floor <NUM>, wall <NUM> and wall <NUM>. Pictures <NUM> and <NUM> are shown on wall <NUM> and a desktop inlay 302a is shown on desk <NUM>. Scene <NUM> may be a scene of an actual room or a scene of a reduced scale model of a room. For example, scene <NUM> may show a view of an architectural model. <FIG> also shows selection apparatus <NUM> of <FIG> comprising tip <NUM> which would be held in the hand of user <NUM> and manipulated as user <NUM> views the scene <NUM>. In another scenario selection apparatus <NUM> may be held and manipulated by a person other than user <NUM>. This may be useful when the view frustum of device <NUM> extends beyond the reach of user <NUM>.

In an example scenario, user <NUM> may select points, surfaces or objects in scene <NUM> by tapping the tip <NUM> of selection apparatus <NUM> on a subject. For example, user <NUM> may select desk inlay 302a by tapping desk inlay 302a with tip <NUM>. The processes of <FIG> may then be used to accurately determine the location of the tap point on desk inlay 302a by determining the location of tip <NUM> in response to the tap. Application programs in device <NUM> may then determine that the subject of the tap event is desk inlay 302a. The implementation of providing an accurate tap location may have particular advantages, for example, when the tap is performed near the border between desk inlay 302a and desk <NUM>. It may be difficult to determine which of desk inlay 302a or desk <NUM> is selected if other selection methods were used. In this case the availability of an accurate tap location may resolve ambiguities in determining which object is the selected subject. The desk inlay 302a may then be manipulated as desired by AR programs in device <NUM>. For example, when it is determined that desk inlay 302a is the subject of the tap event, different colors may be overlaid and sampled on the desk inlay 302a. In another example, an object such as ashtray <NUM> may be selected by tapping it with tip <NUM>. In a situation in which ashtray <NUM> is small and of a similar color as desk inlay 302a, an accurate location of a tap event on ashtray <NUM> may allow device <NUM> to determine that ashtray <NUM> has been selected. In certain situations, it may also provide data to device <NUM> that ashtray <NUM> exists in scene <NUM> and needs to be further processed for the <NUM>-D map if device <NUM> did not initially recognize ashtray <NUM>. In other example use scenarios, accurate tap event location determination may allow accurate object or surface selection related to other details of scene <NUM>. For example, tap event location using stylus <NUM> and tip <NUM> may allow the selection of different portions of pictures <NUM> or <NUM> without ambiguity. In example scenarios, accurate tap event location may allow user <NUM> to accurately select a frame of picture <NUM> or <NUM> as opposed to the whole picture, or select an individual object depicted in the picture. Accurate tap event location of the embodiments may also be used to accurately select points in scene <NUM> so that virtual objects placed into scene <NUM> by device <NUM> may be precisely positioned. For example, user <NUM> may tap a point on wall <NUM> or <NUM> and device <NUM> may position a virtual object relative to the location of the tap event. In another example, a virtual object placed into scene <NUM> may comprise an object such as a virtual keyboard or control device having keys or selection buttons projected onto a surface such as desk inlay 302a. User <NUM> may use selection apparatus <NUM> to select a particular virtual key or selection button as a tap event subject according to the operations of <FIG> by tapping the key or button's image on desk inlay 302a. In a further implementation, when an accelerometer is used in selection apparatus <NUM> as tap sensor <NUM>, a virtual object may be selected as the tap event subject. For example, picture <NUM> may be a virtual picture inserted into scene <NUM> by device <NUM>. Picture <NUM> may then be selected by virtually tapping picture <NUM> using rapid de-acceleration or cessation of movement of selection device <NUM> at the edge of picture <NUM>. Selection device <NUM> may then determine from the de-acceleration or cessation of movement that a tap event which selected picture <NUM> has occurred.

In another example implementation, device <NUM> and selection apparatus <NUM> may be configured so that a tap event initiates a continuous contact with a surface. This implementation may be configured with pressure sensing at tip <NUM> using tap sensor <NUM> of selection apparatus <NUM> to monitor the continuous contact. An application in device <NUM> may be configured for determining and processing a continuous contact of the selection apparatus <NUM> with a surface. In example of this implementation, the application in device <NUM> that processes continuous contact of selection apparatus <NUM> with a surface may be activated by user <NUM> as desired. In this implementation, the location of the tap event is determined by determining the location of the tip <NUM> of selection apparatus <NUM> during an initial tap on a surface. Then the location of the tip <NUM> is continuously determined during the continuous contact of tip <NUM> with the surface. The determining and processing of a continuous contact may be used, for example, to trace accurate <NUM> dimensional (<NUM>-D) shapes to model surfaces in a view frustum. For example, in scenario <NUM> of <FIG> an accurate <NUM>-D model of the surface of desk inlay 302a may be constructed by tapping the edge of inlay 302a, moving selection device <NUM> to trace around the edge of inlay 302a, and continuously determining and recording the location of tip <NUM> throughout the tracing. This would provide an accurate <NUM>-D model of inlay 302a that would be especially useful if inlay 302a was not completely square, or desk <NUM> and inlay 302a were both warped or of irregular shape along the edges of inlay 302a. The <NUM>-D model could then be used to build a replacement for inlay 302a that would accurately fit in the desk <NUM>.

Referring now to <FIG>, therein is illustrated an example of use of an implementation from within the view frustum of a device using a selection apparatus for surface location. <FIG> illustrates a scene <NUM> in a view frustum of a device. The view frustum may be, for example, a view frustum of device <NUM> of <FIG>. In an example implementations using the tap event location of <FIG>, device <NUM> determine <NUM>-D map of scene <NUM> relative to a reference coordinate frame represented by the axis shown as lines <NUM>, <NUM> and <NUM>. Scene <NUM> illustrates a view of a corner of a room having floor <NUM>, wall <NUM> and wall <NUM>. The <NUM>-D map of scene <NUM> has been created by device <NUM> to model surfaces on floor <NUM>, wall <NUM> and wall <NUM>, respectively, using <NUM>-D meshes <NUM>, <NUM> and <NUM>. In the scenario shown in <FIG> accurate tap event location may be used to refine or confirm the <NUM>-D mesh data that models a particular surface. For example, user <NUM> may perform tapping at points <NUM>, <NUM> and <NUM> on the actual physical wall <NUM> using selection apparatus <NUM>. For each tap, device <NUM> and selection apparatus <NUM> may determine the location of each tap event according to the process of <FIG>. The location of the tap events on the actual physical wall at points <NUM>, <NUM> and <NUM> in the <NUM>-D map of scene <NUM> may then be compared with the location data associated with <NUM>-D mesh <NUM>. The location data associated with the <NUM>-D mesh may then be modified or adjusted to be consistent with the location data obtained from the tap events to create a more accurate model of the surface of wall <NUM>. If wall <NUM> is warped or curved a more accurate <NUM>-D mesh model may be obtained. The implementation of <FIG> has use, for example, when a scene includes irregular surfaces with variations, and a device, such as device <NUM>, is not able to accurately create a <NUM>-D mesh to model the surface.

Referring now to <FIG>, therein are illustrated points and surfaces in an example view frustum of a device for use in performing measurements according to an implementation. <FIG> illustrates a scene including wall <NUM> and wall <NUM>. In the example, wall <NUM> and wall <NUM> may be walls or sides of a structure that are situated substantially parallel to each other. The view frustum may be, for example, a view frustum of device <NUM> of <FIG> modeling wall <NUM> and wall <NUM> in a <NUM>-D map. The structure may be, for example, a room, a piece of furniture, or any other structure of different size.

Referring now to <FIG>, therein is a flow diagram illustrating example operations performed when measuring between points in <FIG> illustrates an example of how the distance between wall <NUM> and wall <NUM> may be measured using tap event detection and location according to <FIG>. The process of <FIG> begins at <NUM> where a measurement application is activated on device <NUM>. At <NUM>, device <NUM> receives an indication that a tap event has occurred from selection apparatus <NUM> when a user of selection apparatus <NUM> taps at point <NUM> on surface <NUM> of wall <NUM>. Device <NUM> then determines the location of point <NUM> by determining the location of fiducial marker <NUM> on selection apparatus <NUM> relative to device <NUM>. At <NUM>, device <NUM> receives an indication that a second tap event has occurred from selection apparatus <NUM> when a user of selection apparatus <NUM> taps at point <NUM> on surface <NUM> of wall <NUM>. Device <NUM> then determines the location of point <NUM> by determining the location of fiducial marker <NUM> on selection apparatus <NUM> relative to device <NUM>. At <NUM>, device <NUM> uses the locations of point <NUM> and point <NUM> relative to device <NUM> to determine the locations of point <NUM> and point <NUM> relative to the coordinate frame of the <NUM>-D map. Then at <NUM>, device <NUM> determines the distance between point <NUM> and point <NUM> using the <NUM>-D map to measure the distance between wall <NUM> and wall <NUM> at line <NUM>.

Referring now to <FIG>, therein is a flow diagram illustrating example operations performed when measuring between surfaces of <FIG> according to another implementation using the tap event location of <FIG>. The process of <FIG> begins at <NUM> where a measurement application is activated on device <NUM>. At <NUM>, device <NUM> determines the <NUM>-D geometry and a semantic labeling of objects in its view frustum to create a <NUM>-D map. Depending on use context and the measurement application, the semantic labeling may be determined and applied by recognizing surfaces <NUM> and <NUM> as surfaces of wall <NUM> and wall <NUM>, respectively. At <NUM>, device <NUM> receives and indication that a tap event has occurred from selection apparatus <NUM> when a user of selection apparatus <NUM> taps at point <NUM> on surface <NUM> of wall <NUM>. Device <NUM> then determines the location of point <NUM> by determining the location of fiducial marker <NUM> on selection apparatus <NUM> relative to device <NUM> upon the tap event at point <NUM>. At <NUM>, device <NUM> receives an indication that a second tap event has occurred from selection apparatus <NUM> when a user of selection apparatus <NUM> taps at point <NUM> on surface <NUM> of wall <NUM>. Device <NUM> then determines the location of point <NUM> by determining the location of fiducial marker <NUM> on selection apparatus <NUM> relative to device <NUM>. At <NUM>, device <NUM> determines, from the semantic labeling, that the tap event at point <NUM> selected surface <NUM> of wall <NUM>. At <NUM>, device <NUM> determines, from the semantic labeling, that the tap event at point <NUM> selected surface <NUM> of wall <NUM>. At <NUM>, the measurement application program in device <NUM> then uses the <NUM>-D map to measure the distance between surface <NUM> and surface <NUM> by determining the distance along line <NUM> which is perpendicular to each of selected surface <NUM> and <NUM>. The measurement process of <FIG> may provide more accuracy than the measurement process of <FIG>. This will be the case when a user of selection apparatus <NUM> is not certain that the selected tap event points in <FIG> are directly across from one another or a user cannot reach points directly across from one another on the walls. For example, if the implementation of <FIG> was used with tap event points <NUM> and <NUM>, the distance between surfaces <NUM> and <NUM> would be measured inaccurately as the distance shown by dotted line <NUM>.

Referring now to <FIG>, therein is illustrated another example of a selection apparatus. <FIG> illustrates selection apparatus <NUM> as an example stylus shaped apparatus. The selection apparatus <NUM> may be one configuration of the selection apparatus <NUM> illustrated in <FIG>. Selection apparatus <NUM> comprises a case portion <NUM> which may include circuitry/processors and mechanical structure configured to implement functions according to the embodiments. Case portion <NUM> comprises a tip <NUM> and fiducial marker <NUM> that is disposed on case portion <NUM> adjacent to tip <NUM>. In an implementation, fiducial marker <NUM> may comprise a portion of tip <NUM> or be incorporated within tip <NUM>. Fiducial marker <NUM> may comprise, for example, a retroreflective material or colored LED.

<FIG> is a simplified block diagram illustrating portions of example selection apparatus <NUM> of <FIG>. Selection apparatus <NUM> includes tap sensor <NUM>, accelerometer <NUM>, communication module <NUM>, processor <NUM>, sound generator <NUM>, control inputs <NUM>, and memory <NUM>. Memory <NUM> includes tap detection and processing programs <NUM> and communication control programs <NUM>. In implementations of selection apparatus <NUM>, selection apparatus <NUM> may comprise more or less functions then those illustrated by the blocks of <FIG>. The configuration of functions in an implementation depends on the operations used for determining that a tap event has occurred and the operations used for generating an indication of the tap event to another device such as device <NUM> of <FIG>.

Tap sensor <NUM> determines when a tap event occurs at tip <NUM>. Tap sensor <NUM> may be, for example, a pressure sensing device or a switch at tip <NUM> that is activated by a tap event. Accelerometer <NUM> senses and generates movement data associated with selection apparatus <NUM> that may also be used by processor <NUM> in determining when tap events occur. In implementations of selection apparatus <NUM>, tap sensor <NUM> and accelerometer <NUM> may be used in conjunction to sense tap events and provide data to be processed by processor <NUM> to determine when tap events occur. In alternative implementations, either of accelerometer <NUM> or tap sensor <NUM> may be implemented without the other in selection apparatus <NUM> to provide tap event data to be processed by processor <NUM> to determine when tap events occur.

Communication module <NUM> communicates an indication of a tap event to another device such as device <NUM> of <FIG>. Communication module <NUM> also may be used to communicate any other control/setup information between selection apparatus <NUM> and another device such as device <NUM>. Communication module <NUM> may be, for example, a short range wireless module configured to communicate signals using a protocol such as Bluetooth, Wi-Fi, Wi-Fi Direct, or any other wireless protocol that may be used to communicate with another device such as device <NUM>. Sound generator <NUM> may be any type of device that generates a unique click or sound that is detectable at another device such as device <NUM>.

Memory <NUM> may be implemented as any type of computer readable storage media in selection apparatus <NUM>, including non-volatile and volatile memory. Processing unit <NUM> may comprise one or more processors, or other control circuitry or any combination of processors and control circuitry. In an implementation, when executed, tap detection and processing programs <NUM> and communications control programs <NUM> cause processor <NUM> to control selection apparatus <NUM> to implement tap event determination and generation of tap event indications to another device by performing the operations of <FIG>.

Selection apparatus <NUM> also may include control inputs <NUM> for setup/control of selection apparatus <NUM>. Control inputs <NUM> may comprise one or more switches or buttons on selection apparatus <NUM> that allow a user to set a mode, for example, a mode for using communication module or a mode for using sound generator <NUM>, to generate an indication of a tap event depending on the communication capabilities of other devices with which selection apparatus <NUM> communicates. Control inputs <NUM> may also be used to allow a user to set a mode for tap detection, for example, a mode that uses an accelerometer <NUM> or a tap sensor <NUM>, or both, to determine a tap event has occurred. In an alternative implementation, for example, one or more of control inputs <NUM> may be incorporated into communication module <NUM> and control/setup of selection apparatus <NUM> may be performed wirelessly.

Referring now to <FIG>, therein is a simplified block diagram illustrating portions of an example device implemented according to the disclosed embodiments. Device <NUM> represents a possible implementation of devices, such as device <NUM> of <FIG>. Device <NUM> may include a display and input/output (I/O) <NUM> that provides an interface for a user. Display and I/O <NUM> may include, for example, a touch screen or keypad, a microphone, speakers, a visual display, a virtual display created in the view frustum of device <NUM>, or other I/O functions which receive inputs and provide outputs to and from a user to allow the user to control device <NUM>. Device <NUM> includes processor <NUM> and memory/storage <NUM> which is shown as including computer program code or instructions for tap detection and location programs <NUM>, operating system code (OS) <NUM>, and application programs <NUM>. Device <NUM> also includes wireless communication module <NUM> that is used to receive indications of occurrences of tap events from a selection apparatus such as selection apparatus <NUM>. Wireless communication module <NUM> may also be used to exchange other control/setup information with the selection apparatus.

<FIG> also shows device <NUM> as including sensors comprising depth camera <NUM>, RGB camera <NUM>, inertial measurement unit (IMU) <NUM>, and her internal sensors <NUM>. Depth camera <NUM>, RGB camera <NUM>, inertial measurement unit (IMU) <NUM> and internal sensors <NUM> provide the data used for creating a <NUM>-D map of a scene in the view frustum of device <NUM> relative to a reference coordinate frame and for self-tracking the location and orientation of device <NUM> in the reference coordinate frame. Memory <NUM> may be used to store data generated during self-tracking of the location of device <NUM> and the <NUM>-D map data. Depth camera unit <NUM>, RGB camera unit <NUM>, inertial measurement unit (IMU) <NUM> and internal sensors <NUM> may also provide the data used to determine the location of a tap event, for example, by determining the location of fiducial marker <NUM> on selection apparatus <NUM>, relative to device <NUM> upon receiving an indication of a tap event.

In an implementation, when executed, tap detection and location programs <NUM> cause processor <NUM> to control device <NUM> to implement tap detection and tap location by performing one or more operations of <FIG>. In particular implementations, application programs <NUM> may cause processor to control device <NUM> to implement operations such as operation <NUM> and <NUM> of <FIG> to process tap event occurrence and location data. In other example implementations, when executed, tap detection and location programs <NUM> and application programs <NUM> may also perform operations described in relation to <FIG> or <FIG>.

Memory <NUM> may be implemented as any type of computer readable storage media in device <NUM>, including non-volatile and volatile memory. Processing unit <NUM> may comprise one or more processors, or other control circuitry or any combination of processors and control circuitry. Processing unit <NUM> provides overall control of device <NUM> and the other functional blocks shown in <FIG> by executing instructions or code in memory <NUM> to provide necessary functions for operation of device <NUM> according to the disclosed embodiments.

The functions shown in <FIG> may be implemented in a device such as device <NUM> of <FIG>, a dedicated camera device or in a device, such as a laptop computer, a tablet computing device or smartphone that also has other capabilities in addition to the illustrated functions. Any device with the capability to determine/receive the self-tracking and tap event location data and/or process the data may be used to implement the embodiments. Various implementations of devices, according to the embodiments may include more or less functions than those shown in <FIG>.

For example, implementations of device <NUM> may include implementations as a smart phone, a tablet computer, camera, video camera, AR devices, a desktop computer, laptop computer device, gaming devices, media devices, smart televisions, home theater systems, smart automobile systems, smart house systems, multimedia cable/television boxes, smart phone accessory devices, tablet accessory devices, personal digital assistants (PDAs), portable media players, smart sensors, multimedia studio systems, or industrial control systems. Certain of the functional blocks shown in <FIG> may be omitted, added to, combined, or rearranged in these other implementations.

The example embodiments disclosed herein may be described in the general context of processor-executable code or instructions stored on memory that may comprise one or more computer readable storage media (e.g., tangible non-transitory computer-readable storage media such as memory <NUM> or <NUM>). As should be readily understood, the terms "computer-readable storage media" or "non-transitory computer-readable media" include the media for storing of data, code and program instructions, such as memory <NUM> or <NUM>, and do not include portions of the media for storing transitory propagated or modulated data communication signals.

The disclosed embodiments include a device comprising one or more processors and memory in communication with the one or more processors, the memory comprising code that, when executed, causes the one or more processors to receive data associated with a scene in the view frustum of the device from at least one sensor, determine a <NUM>-dimensional map of the scene in the view frustum relative to a coordinate frame based the data, receive an indication from a selection apparatus that a tap event has occurred within the scene in the view frustum of the device, locate, in response to the tap event occurring, at least a portion of the selection apparatus in the view frustum relative to the device using the at least one sensor, determine a location of the tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus, and, determine a subject of the tap event based on the location of the tap. The subject may comprise a real object within the scene in the view frustum of the device or the subject may comprise a virtual object within the scene in the view frustum of the device.

The <NUM>-dimensional map may comprise location data associated with a <NUM>-dimensional mesh that represents at least a portion of a surface, and the device may determine the subject by determining the at least a portion of the at least one surface based on the location of the tap event and the location data associated with the <NUM>-dimensional mesh. The code, when executed, may further cause the one or more processors to adjust the location data associated with the <NUM>-dimensional mesh based on the location of the tap event on the surface. The tap event may comprise a first tap event, the subject may comprise a first point, the indication received from the selection apparatus may comprise a first indication received from the selection apparatus, and the code, when executed, may further cause the one or more processors to determine that a second tap event has occurred based on a second indication received from the selection apparatus, locating, in response to the second tap event occurring, at least a portion of the selection apparatus in the view frustum relative to the device using the at least one sensor, determine a location of the second tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus located in response to the second tap event occurring, determine a second point based on the location of the second tap event, and, determine a distance between the first and second point.

Embodiments are also disclosed in which the subject comprises a first subject, the <NUM>-dimensional map comprises location data associated with a first and a second <NUM>-dimensional mesh that represent, respectively, the first and a second subject in the scene in the view frustum device, the tap event comprises a first tap event, the indication received from the selection apparatus comprise a first indication received from the selection apparatus, and the code, when executed, causes the one or more processors to determine the first subject from the first <NUM>-dimensional mesh based on the location of the first tap event, determine that a second tap event has occurred based on a second indication received from the selection apparatus, locate, in response to the second tap event occurring, at least a portion of the selection apparatus in the view frustum relative to the device using the at least one sensor, determine a location of the second tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus located in response to the second tap event occurring, determine the second subject from the second <NUM>-dimensional mesh based on the location of the second tap event, and, determine a distance between the first and second subject based on the <NUM>-dimensional map.

The tap event may initiate a continuous contact with the subject and the code, when executed, may cause the one or more processors to determine the location of the selection apparatus relative to the coordinate frame during the continuous contact. Further embodiments of a device are also disclosed in which the indication received from the selection apparatus comprises data from an accelerometer in the selection apparatus, the indication received from the selection apparatus data comprises data from a pressure sensor in the selection apparatus. The device may determine the location of the tap event by determining a location of a fiducial marker on the selection apparatus.

Disclosed embodiments also include a method comprising receiving data associated with a scene in the view frustum of a device from at least one sensor, determining a <NUM>-dimensional map of the scene in the view frustum of the device relative to a coordinate frame based on the data, receiving an indication from a selection apparatus that a tap event has occurred within the scene in the view frustum of the device, locating, in response to the tap event occurring, at least a portion of the selection apparatus in the view frustum relative to the device using the at least one sensor, determining a location of the tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus, and, determining a subject of the tap event based on the location of the tap event. The subject may comprise a real object in the scene in the view frustum of the device or the subject may comprise a virtual object in a scene in the view frustum of the device. The <NUM>-dimensional map may comprise location data associated with a <NUM>-dimensional mesh that represents at least a portion of a surface, and the determining the subject of the tap event may comprise determining the at least a portion of the surface based on the location of the tap event and the location data associated with the <NUM>-dimensional mesh. The method may further comprise adjusting the location data associated with the <NUM>-dimensional mesh based on the location of the tap event on the at least a portion of a surface.

Embodiments of the method are also disclosed in which the tap event comprises a first tap event, the subject comprises a first point, the indication received from the selection apparatus comprise a first indication received from the selection apparatus, and the method further comprises receiving a second indication from the selection apparatus that a second tap event has occurred in the scene in the view frustum of the device, locating, in response to the second tap event occurring, at least a portion of the selection apparatus in the view frustum relative to the device using the at least one sensor, determining a location of the second tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus located in response to the second tap event occurring, determining a second point based on the location of the second tap event, and, determining a distance between the first and second points. Embodiments of the method also those in which the subject comprises a first subject, the <NUM>-dimensional map comprises location data associated with a first and a second <NUM>-dimensional mesh that represent, respectively, the first and a second subject in the scene in the view frustum of the device, the tap event comprises a first tap event, the indication received from the selection apparatus comprise a first indication received from the selection apparatus, and the method further comprises determining the first subject from the first <NUM>-dimensional mesh based on the location of the first tap event, receiving an second indication from the selection apparatus that a second tap event has occurred in the scene in the view frustum of the device, locating, in response to the second tap event occurring, at least a portion of the selection apparatus in the view frustum relative to the device using the at least one sensor, determining a location of the second tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus located in response to the second tap event occurring, determining the second surface from the second <NUM>-dimensional mesh based on the location of the second tap event, and, determining a distance between the first and second subject based on the <NUM>-dimensional map.

The disclosed embodiments also include an apparatus comprising a fiducial marker to visually indicate a location of the apparatus to a computing device when the location is within a view frustum of the computing device, a tap event sensor for detecting a tap event comprising a contact of the apparatus with a subject remote to the computing device, and, a module for communicating an indication of the occurrence of the tap event to the computing device. The tap event sensor may comprise an accelerometer. The apparatus may comprise a stylus having a casing including a tip, the fiducial maker may be disposed at the tip, the tap event sensor may be included within the casing and coupled to the tip of the stylus, and the module may be included within the casing.

While implementations have been disclosed and described as having functions implemented on particular devices, one or more of the described functions for the devices may be moved between the devices and implemented on a different one of the devices than shown in the figures, or on different types of equipment.

While the functionality disclosed herein has been described by illustrative example using descriptions of the various components and devices of embodiments by referring to functional blocks and processors or processing units, controllers, and memory including instructions and code, the functions and processes of the embodiments may be implemented and performed using any type of processor, circuitry or combinations of processors and/or circuitry and code. This may include, at least in part, one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Use of the term processor or processing unit in this disclosure is meant to include all such implementations.

Claim 1:
An augmented reality device (<NUM>, <NUM>) comprising:
at least one sensor for sensing and receiving data associated with a scene in a view frustrum of the device;
a communication module (<NUM>);
one or more processors (<NUM>) in communication with the at least one sensor; and a memory (<NUM>, <NUM>) in communication with the one or more processors, the memory comprising code that, when executed, causes the one or more processors to control the augmented reality device (<NUM>, <NUM>) to:
receive data (<NUM>, <NUM>, <NUM>) suitable for determining a <NUM>-dimensional map and associated with a scene (<NUM>, <NUM>) in the view frustum using the at least one sensor;
determine a <NUM>-dimensional map of the scene (<NUM>, <NUM>) in the view frustum relative to a coordinate frame based on the data (<NUM>, <NUM>, <NUM>);
receive, using the communication module (<NUM>), an indication comprising a wireless signal (<NUM>) from a selection apparatus (<NUM>, <NUM>) that a tap event (<NUM>, <NUM>, <NUM>) has occurred;
locate, in response to the tap event (<NUM>, <NUM>, <NUM>) occurring, at least a portion of the selection apparatus (<NUM>, <NUM>) in the view frustum relative to the augmented reality device (<NUM>, <NUM>) using the at least one sensor; determine a location of the tap event relative to the coordinate frame based on the location of the at least a portion of the selection apparatus; and,
determine a target subject of the tap event (<NUM>, <NUM>, <NUM>) based on the location of the tap event.