Patent Publication Number: US-10769437-B2

Title: Adaptive sampling of training views

Description:
BACKGROUND 
     1. Technical Field 
     The disclosure relates generally to the field of training object detection algorithms, and more specifically to methods and systems for training object detection algorithms using synthetic two-dimensional (2D) images. 
     2. Related Art 
     Augmented Reality (AR) has become increasingly common with the advancement of computer technology. A general definition of AR is capturing a real-world scene and adding artificial (virtual) elements using software. This can enhance a user&#39;s perception of the real world or provide entertainment to the user. 
     Object tracking is used in many AR implementations. In object tracking, a real-world object is “followed” by an artificial object, such as computer-graphics enhancements or an information bubble (as perceived by the user). In other words, if the real-world object moves or the user&#39;s view moves relative to the real-world object, the artificial object will remain in the same location relative to the real-world object and/or the content of the artificial object will be determined based on the movement and/or new location of the real-world object. Location tracking is also used in many AR implementations. In location tracking, a virtual object will stay in one location in the scene, regardless of the movement of the user. 
     One platform for implementing AR is the smartphone. The presence of a camera, display device, and processor on the same device allows for software to easily add artificial elements to a live scene captured by the camera. Moreover, the presence of motion sensors and locators (e.g. accelerometers and GPS) on these devices is exploited by the software to better implement AR. 
     Although smartphones provide a simple and convenient platform for implementing AR, they do not provide a very immersive experience for the user. This is because the user&#39;s eyes are spatially separated from the smartphone, and instead of perceiving the environment with their own eyes, they are viewing the scene as captured by the camera. 
     SUMMARY 
     To improve on the AR experience, the transparent head-mounted display (HMD) can implement AR. These systems are usually glasses with prisms placed in front of the eyes. The user views the scene directly through the glasses. The prisms allow for artificial images to be overlaid on the scene as perceived by the user. Meanwhile, the HMD collects data from the scene using a camera. 
     3D pose estimation is an important technology with many applications, including the fields of AR, VR and robotics. 
     Trackers commonly utilize an initialization method to first start tracking and to re-start tracking in case of tracking loss. This may require estimating the pose of an object from an image without prior history, a technology sometimes referred to as object detection and pose estimation (ODPE). 
     Object detection algorithms are commonly trained to recognize specific objects using images of the object captured with a camera that is to be used for the AR system. In at least some known systems, the initial stages of the training process are time consuming and performed manually. In such manual training, a trainer positions the object, captures numerous images of the object from numerous different angles of view using the camera of the AR system, and uploads the images to a training computer. Using the training computer, the trainer aligns a three-dimensional (3D) model of the object to the image of the object in each captured image. 
     An advantage of some aspects of the disclosure is to solve at least a part of the problems described above, and aspects of the disclosure can be implemented as the following aspects. 
     One aspect of the disclosure is a non-transitory computer-readable medium that, when executed by one or more processors, performs a method, which comprises a step of receiving, in a memory, feature data sets of (i) a reference object or (ii) a 3D model. Each image feature set is obtained from an image captured or generated from a primary set of views within a view range around the reference object or 3D model. The method also includes the step of deriving similarity scores between each pair of adjacent views among the primary set of views using the feature data sets and sampling a secondary set of views between the adjacent primary views in the case where the similarity score is equal to or less than a threshold so that sampling density varies across the view range. The method also includes generating the training data corresponding to the primary set of views and the secondary set of views. The method may also include sampling the views at a coarse sampling resolution at equal angles from each other with respect to azimuth and/or elevation and storing the sampled views in the memory. Similarity scores may be based on appearance and/or shape of adjacent views of the reference object or 3D model. 
     Another aspect of the disclosure is a method comprising the steps of storing, in a memory, multiple views of an object, deriving similarity scores between adjacent views, and varying a sampling density based on the similarity scores. 
     A further aspect of the disclosure is a head-mounted display device. The head-mounted display device includes memory configured to store training data used for detecting an object. The training data is derived from multiple views of the object. The head-mounted display device also includes a processing device configured to utilize the training data to detect the object and a display component configured to display CAD images of the detected object over a user&#39;s view of the object. Before object detection, the multiple views are obtained using a varying sampling density based on similarity scores of adjacent views. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The disclosure will be described with reference to the accompanying drawings, wherein like numbers reference like elements. 
         FIG. 1  is a diagram illustrating a schematic configuration of an embodiment of a head-mounted display (HMD). 
         FIG. 2  is a block diagram illustrating an embodiment of a functional configuration of the HMD shown in  FIG. 1 . 
         FIG. 3  is a block diagram illustrating an embodiment of a functional configuration of a computer for performing the methods of this disclosure. 
         FIG. 4  is a flow diagram illustrating an embodiment of a method according to this disclosure. 
         FIG. 5  is a diagram illustrating an embodiment of an input window of a graphic user interface (GUI) for use with exemplary methods of this disclosure. 
         FIG. 6  is a diagram illustrating an embodiment of a preview window associated with the GUI input window shown in  FIG. 5 . 
         FIGS. 7A, 7B, and 7C  illustrate flow diagrams illustrating an embodiment of a method for adaptive sampling according to this disclosure. 
         FIGS. 8A, 8B, 8C and 8D  are diagrams illustrating examples of sampling locations according to the method of  FIG. 7 . 
         FIGS. 9A, 9B, and 9C  are diagrams illustrating an example of samples obtained using the sampling method of  FIG. 7 . 
         FIG. 10  is a diagram illustrating various views around an object. 
     
    
    
     DESCRIPTION OF EXEMPLARY EMBODIMENTS 
     The disclosure relates generally to training object detection algorithms, and more specifically to methods and systems for training object detection algorithms using synthetic two-dimensional (2D) images. 
     In some embodiments, the trained object detection algorithm is used by an object detection device, such as an AR device. Some example systems include and/or interface with an AR device. In still other embodiments, the methods described herein for training an object detection algorithm are performed by the AR device itself. 
     The AR device may be, for example, a head-mounted display (HMD). An example HMD suitable for use with the methods and systems described herein will be described with reference to  FIGS. 1 and 2 . 
       FIG. 1  is a schematic configuration of an HMD  100 . The HMD  100  is a head-mounted display device (a head mounted display). The HMD  100  is an optical transmission type. That is, the HMD  100  can cause a user to sense a virtual image and, at the same time, cause the user to directly visually recognize an outside scene. 
     The HMD  100  includes a wearing belt  90  wearable on the head of the user, a display section  20  that displays an image, and a control section  10  that controls the display section  20 . The display section  20  causes the user to sense a virtual image in a state in which the display section  20  is worn on the head of the user. The display section  20  causing the user to sense the virtual image is referred to as “display AR” as well. The virtual image sensed by the user is referred to as AR image as well. 
     The wearing belt  90  includes a wearing base section  91  made of resin, a belt  92  made of cloth coupled to the wearing base section  91 , a camera  60 , and an IMU (Inertial Measurement Unit)  71 . The wearing base section  91  has a shape curved along the form of the frontal region of a person&#39;s forehead. The belt  92  is worn around the head of the user. 
     The camera  60  functions as an imaging section. The camera  60  is capable of imaging an outside scene and disposed in a center portion of the wearing base section  91 . In other words, the camera  60  is disposed in a position corresponding to the center of the forehead of the user in a state in which the wearing belt  90  is worn on the head of the user. Therefore, the camera  60  images an outside scene, which is a real scene on the outside in a line of sight direction of the user, and acquires a captured image, which is an image captured by the camera  60 , in the state in which the user wears the wearing belt  90  on the head. 
     The camera  60  includes a camera base section  61  that rotates with respect to the wearing base section  91  and a lens section  62 , a relative position of which is fixed with respect to the camera base section  61 . The camera base section  61  is disposed to be capable of rotating along an arrow CS 1 , which indicates a predetermined range of an axis included in a plane including the center axis of the user, when the wearing belt  90  is worn on the head of the user. Therefore, the direction of the optical axis of the lens section  62 , which is the optical axis of the camera  60 , can be changed in the range of the arrow CS 1 . The lens section  62  images a range that changes according to zooming centering on the optical axis. 
     The IMU  71  is an inertial sensor that detects acceleration. The IMU  71  can detect angular velocity and terrestrial magnetism in addition to the acceleration. The IMU  71  is incorporated in the wearing base section  91 . Therefore, the IMU  71  detects acceleration, angular velocity, and terrestrial magnetism of the wearing belt  90  and the camera base section  61 . 
     A relative position of the IMU  71  to the wearing base section  91  is fixed. Therefore, the camera  60  is movable with respect to the IMU  71 . Further, a relative position of the display section  20  to the wearing base section  91  is fixed. Therefore, a relative position of the camera  60  to the display section  20  is movable. In some other embodiments, the camera  60  and IMU  71  may be provided in the display section  20 , so that they are fixed with respect to the display section  20 . The spatial relationships represented by the rotation and translation matrices among the camera  60 , IMU  70 , and display section  20  (including right/left optical-image display sections  26 / 28 ), which have been obtained by calibration, are stored in a memory area or device in the control section  10 . 
     The display section  20  is coupled to the wearing base section  91  of the wearing belt  90 . The display section  20  is an eyeglass type. The display section  20  includes a right holding section  21 , a right display driving section  22 , a left holding section  23 , a left display driving section  24 , a right optical-image display section  26 , and a left optical-image display section  28 . 
     The right optical-image display section  26  and the left optical-image display section  28  are located in front of the right eye and the left eye of the user when the user wears the display section  20 . One end of the right optical-image display section  26  and one end of the left optical-image display section  28  are connected to each other in a position corresponding to the middle of the forehead of the user when the user wears the display section  20 . 
     The right holding section  21  has a shape extending in a substantial horizontal direction from an end portion ER, which is the other end of the right optical-image display section  26 , and inclining obliquely upward halfway. The right holding section  21  connects the end portion ER and a coupling section  93  on the right side of the wearing base section  91 . 
     Similarly, the left holding section  23  has a shape extending in a substantial horizontal direction from an end portion EL, which is the other end of the left optical-image display section  28  and inclining obliquely upward halfway. The left holding section  23  connects the end portion EL and a coupling section (not shown in the figure) on the left side of the wearing base section  91 . 
     The right holding section  21  and the left holding section  23  are coupled to the wearing base section  91  by left and right coupling sections  93  to locate the right optical-image display section  26  and the left optical-image display section  28  in front of the eyes of the user. Note that the coupling sections  93  couple the right holding section  21  and the left holding section  23  to be capable of rotating and capable of being fixed in any rotating positions. As a result, the display section  20  is provided to be capable of rotating with respect to the wearing base section  91 . 
     The right holding section  21  is a member provided to extend from the end portion ER, which is the other end of the right optical-image display section  26 , to a position corresponding to the temporal region of the user when the user wears the display section  20 . 
     Similarly, the left holding section  23  is a member provided to extend from the end portion EL, which is the other end of the left optical-image display section  28  to a position corresponding to the temporal region of the user when the user wears the display section  20 . The right display driving section  22  and the left display driving section  24  are disposed on a side opposed to the head of the user when the user wears the display section  20 . 
     The display driving sections  22  and  24  include liquid crystal displays  241  and  242  (hereinafter referred to as “LCDs  241  and  242 ” as well) and projection optical systems  251  and  252  explained below. The configuration of the display driving sections  22  and  24  is explained in detail below. 
     The optical-image display sections  26  and  28  include light guide plates  261  and  262  and dimming plates explained below. The light guide plates  261  and  262  are formed of a light transmissive resin material or the like and guide image lights output from the display driving sections  22  and  24  to the eyes of the user. 
     The dimming plates are thin plate-like optical elements and are disposed to cover the front side of the display section  20  on the opposite side of the side of the eyes of the user. By adjusting the light transmittance of the dimming plates, it is possible to adjust an external light amount entering the eyes of the user and adjust visibility of a virtual image. 
     The display section  20  further includes a connecting section  40  for connecting the display section  20  to the control section  10 . The connecting section  40  includes a main body cord  48  connected to the control section  10 , a right cord  42 , a left cord  44 , and a coupling member  46 . 
     The right cord  42  and the left cord  44  are two cords branching from the main body cord  48 . The display section  20  and the control section  10  execute transmission of various signals via the connecting section  40 . As the right cord  42 , the left cord  44 , and the main body cord  48 , for example, a metal cable or an optical fiber can be adopted. 
     The control section  10  is a device for controlling the HMD  100 . The control section  10  includes an operation section  135  including an electrostatic track pad and a plurality of buttons that can be pressed. The operation section  135  is disposed on the surface of the control section  10 . 
       FIG. 2  is a block diagram functionally showing the configuration of the HMD  100 . As shown in  FIG. 2 , the control section  10  includes a ROM  121 , a RAM  122 , a power supply  130 , the operation section  135 , a CPU  140  (sometimes also referred to herein as processor  140 ), an interface  180 , and a transmitting section  51  (Tx  51 ) and a transmitting section  52  (Tx  52 ). 
     The power supply  130  supplies electric power to the sections of the HMD  100 . Various computer programs are stored in the ROM  121 . The CPU  140  develops or loads, in the RAM  122 , the computer programs stored in the ROM  121  to execute the computer programs. The computer programs include computer programs for realizing tracking processing and AR display processing explained below. 
     The CPU  140  develops, in the RAM  122 , the computer programs stored in the ROM  121  to function as an operating system  150  (OS  150 ), a display control section  190 , a sound processing section  170 , an image processing section  160 , and a processing section  167 . 
     The display control section  190  generates control signals for controlling the right display driving section  22  and the left display driving section  24 . The display control section  190  controls generation and emission of image lights respectively by the right display driving section  22  and the left display driving section  24 . 
     The display control section  190  transmits control signals to a right LCD control section  211  and a left LCD control section  212  respectively via the transmitting sections  51  and  52 . The display control section  190  transmits control signals respectively to a right backlight control section  201  and a left backlight control section  202 . 
     The image processing section  160  acquires an image signal included in contents and transmits the acquired image signal to receiving sections  53  and  54  of the display section via the transmitting sections  51  and  52 . The sound processing section  170  acquires a sound signal included in the contents, amplifies the acquired sound signal, and supplies the sound signal to a speaker (not shown in the figure) in a right earphone  32  and a speaker (not shown in the figure) in a left earphone  34  connected to the coupling member  46 . 
     The processing section  167  acquires a captured image from the camera  60  in association with time. The time in this embodiment may or may not be based on a standard time. The processing section  167  calculates a pose of an object (a real object) according to, for example, a transformation matrix. The pose of the object means a spatial relation (a rotational and a translational relation) between the camera  60  and the object. The processing section  167  calculates, using the calculated spatial relation and detection values of acceleration and the like detected by the IMU  71 , a transformation matrix for converting a coordinate system fixed to the camera  60  to a coordinate system fixed to the IMU  71 . The function of the processing section  167  is used for the tracking processing and the AR display processing explained below. 
     The interface  180  is an input/output interface for connecting various external devices OA, which are supply sources of contents, to the control section  10 . Examples of the external devices OA include a storage device having stored therein an AR scenario, a personal computer (Pc), a cellular phone terminal, and a game terminal. As the interface  180 , for example, a USB interface, a micro USB interface, and an interface for a memory card can be used. 
     The display section  20  includes the right display driving section  22 , the left display driving section  24 , the right light guide plate  261  functioning as the right optical-image display section  26 , and the left light guide plate  262  functioning as the left optical-image display section  28 . The right and left light guide plates  261  and  262  are optical see-through elements that transmit light from real scene. 
     The right display driving section  22  includes the receiving section  53  (Rx 53 ), the right backlight control section  201  and a right backlight  221 , the right LCD control section  211  and the right LCD  241 , and the right projection optical system  251 . The right backlight control section  201  and the right backlight  221  function as a light source. 
     The right LCD control section  211  and the right LCD  241  function as a display element. The display elements and the optical see-through elements described above allow the user to visually perceive an AR image that is displayed by the display elements to be superimposed on the real scene. Note that, in other embodiments, instead of the configuration explained above, the right display driving section  22  may include a self-emitting display element such as an organic EL display element or may include a scan-type display element that scans a light beam from a laser diode on a retina. The same applies to the left display driving section  24 . 
     The receiving section  53  functions as a receiver for serial transmission between the control section  10  and the display section  20 . The right backlight control section  201  drives the right backlight  221  on the basis of an input control signal. The right backlight  221  is a light emitting body such as an LED or an electroluminescence (EL) element. The right LCD control section  211  drives the right LCD  241  on the basis of control signals transmitted from the image processing section  160  and the display control section  190 . The right LCD  241  is a transmission-type liquid crystal panel on which a plurality of pixels is arranged in a matrix shape. 
     The right projection optical system  251  is configured by a collimate lens that converts image light emitted from the right LCD  241  into light beams in a parallel state. The right light guide plate  261  functioning as the right optical-image display section  26  guides the image light output from the right projection optical system  251  to the right eye RE of the user while reflecting the image light along a predetermined optical path. Note that the left display driving section  24  has a configuration same as the configuration of the right display driving section  22  and corresponds to the left eye LE of the user. Therefore, explanation of the left display driving section  24  is omitted. 
     The device to which the technology disclosed as an embodiment is applied may be an imaging device other than an HMD. For example, the device may be an imaging device that has no function of displaying an image. 
       FIG. 3  is a block diagram illustrating a functional configuration of a computer  300  as an information processing device in the present embodiment which performs the methods described herein. The computer  300  includes a CPU  301 , a display unit  302 , a power source  303 , an operation unit  304 , a storage unit  305 , a ROM, a RAM, an AR interface  309  and a network adaptor  310 . The power source  303  supplies power to each unit of the computer  300 . The operation unit  304  is a user interface (GUI) for receiving an operation from a user. The operation unit  304  includes a keyboard, a mouse and a touch pad and the like and their driver software. 
     The storage unit  305  stores various items of data and computer programs, and includes a hard disk drive, a solid-state drive, or the like. The storage unit  305  includes a 3D model storage portion  307  and a template storage portion  308 . The 3D model storage portion  307  stores a three-dimensional model of a target object, created by using computer-aided design (CAD) or other 3D reconstruction methods. The training data storage portion  308  stores training data created as described herein (not shown). The storage unit  305  also stores instructions (not shown) for execution by the CPU  301 . The instructions cause the CPU  301  to perform the methods described herein. The AR interface  309  is an interface for communicative connection to an AR device. The AR interface may be any wired or wireless interface suitable for establishing a data connection for communication between the computer  300  and an AR device. The AR interface may be, for example, a Wi-Fi transceiver, a USB port, a Bluetooth® transceiver, a serial communication port, a proprietary communication port, or the like. The network adaptor  310  is configured to allow CPU  301  to connect to one or more networks to communicate with other computers, such as a server computer via a wireless network, so that, for example, the computer  300  receives from the other computer a computer program that causes the computer  300  to perform functions described in the embodiments described herein. In some embodiments, the AR device interface  309  and the network adaptor  310  are a single adaptor suitable for performing the tasks of both network adaptor  310  and AR device interface  309 . 
     The CPU  301  reads various programs (also sometimes referred to herein as instructions) from the ROM and/or the storage unit  305  and develops the programs in the RAM, so as to execute the various programs. Suitable instructions are stored in storage unit  305  and/or the ROM and executed by the CPU  301  to cause the computer  300  to operate as a training computer to train the object detection algorithm as described herein. In some embodiments, the computer  300 , with the appropriate programming, is a system for training an object detection algorithm using synthetic images. In other embodiments, the HMD  100  is the system for training an object detection algorithm using synthetic images. In still other embodiments, the system for training an object detection algorithm using synthetic images includes the computer  300  and the HMD  100 . 
     The embodiments described herein relate to methods and systems for training an object detection algorithm using synthetic images, rather than actual images of a real-world object. As used herein, synthetic images generally refer to 2D images that are not created using a camera to capture a representation of a 3D scene. More specifically, with respect to training an object detection algorithm to detect a representation of a real-world 3D object in image frames captured by a camera, synthetic images are 2D images that are not created by a camera capturing a representation of the real-world 3D object. Synthetic images may be generated by capturing 2D images of a 3D model of an object in a computer (e.g., a 3D CAD model of an object), drawing (whether by hand or using a computer) a 2D image of the object, or the like. It should be noted that synthetic images include images of a synthetic image. For example, a photograph or scan of a synthetic image may itself be a synthetic image, in one embodiment. Conversely, images of an actual image, such as a photograph or scan of a photograph of the real-world 3D image, may not be synthetic images for purposes of this disclosure under one embodiment. 
       FIG. 4  is a flow diagram of an example method  400  of training an object detection algorithm using synthetic images. The method  400  may be performed by computer  300  to train an object detection algorithm for use with the HMD  100  and will be described with reference to computer  300  and HMD  100 . In other embodiments, the method  400  may be performed by a different computer (including, e.g., the control section  10 ), may be used to train an object detection algorithm for a different AR device, may be used to, and/or may be used to train an object detection algorithm for any other device that performs object detection based on image frames. To facilitate performance by a computer, the method  400  is embodied as instructions executable by one or more processors and stored in a non-transitory computer readable medium. 
     Initially, in S 402 , CPU  301  receives a selection of a 3D model stored in one or more memories, such as the ROM or the storage unit  305 . The 3D model may correspond to a real-world object that the object detection algorithm is to be trained to detect in 2D image frames. In the example embodiment, the selection is received from a user, such as by a user selection through a GUI of the computer  300 . 
     It is noted that a 3D model is discussed herein as being used to generate synthetic images in method  400 . However, in some embodiments, a 3D model may not be required and instead, electronic data other than a 3D model (e.g., a 2D model, one or more 2D or 3D synthetic images, or the like) may be used in step S 402 . As such, for ease of description, the steps of method  400  (and other parts of the present disclosure) are described using a 3D model. However, the present disclosure is not limited to using a 3D model under step S 402  and anywhere where a 3D model is referenced, it should be understood that some embodiments may relate to using electronic data other than a 3D model. 
     A camera parameter set for a camera, such as the camera  60 , for use in detecting a pose of the object in a real scene is set in S 404 . The images captured by different cameras of the same real scene will typically differ at least somewhat based on the particular construction and components of each camera. The camera parameter set defines, at least in part, how its associated camera will capture an image. In the example embodiment, the camera parameter set may include the resolution of the images to be captured by the camera and camera intrinsic properties (or “camera intrinsics”), such as the X and Y direction focal lengths (fx and fy, respectively), and the camera principal points coordinates (cx and cy). Other embodiments may use additional or alternative parameters for the camera parameter set. In some embodiments, the camera parameter set is set by the user, such as by a user selection through a graphical user interface (“GUI”) of the computer  300  (as is discussed later with regard to  FIG. 5 ). 
     In some embodiments, the camera parameter set is set by the computer  300  without being selected by the user. In some embodiments, a default camera parameter set is set by the computer  300 . The default camera parameter set may be used when the camera that will be used in detecting the pose of the object in the real scene is unknown or its parameters are unknown. The default camera set may include the parameters for an ideal camera, a popular camera, a last camera for which a camera parameter set was selected, or any other suitable camera parameter set. Moreover, some embodiments provide a combination of one or more of the above-described methods of setting the camera parameter set. 
     According to various embodiments, the camera parameter set (S 404 ) can be set by many different ways, including by a computer retrieving a pre-stored model from a plurality of models pre-stored on a database, the computer receiving camera parameters from a connected AR device, and/or by a user directly entering (and/or modifying) into a GUI. However, the present application should not be limited to these specific embodiments. Nonetheless, the above embodiments are described herein below. 
     First, in some embodiments, setting the camera parameter set (S 404 ) is performed by receiving information identifying a known AR device including the camera (S 406 ). The information identifying the AR device is received from a user input, such as by selecting, through the computer&#39;s GUI, the AR device from a list of known AR devices. In other embodiments, the user may input the information identifying the AR device, such as by typing in a model name, model number, serial number, or the like. 
     The CPU  301  acquires, based at least in part on the information identifying the AR device, the camera parameter set for the camera (S 408 ). The camera parameter set may be acquired from a plurality of the camera parameter sets stored in one or more memories, such as the storage unit  305  or a local or remote database. Each camera parameter set is associated in the one or more memories with at least one AR device of a plurality of different AR devices. Because multiple different AR devices may include the same camera, a single camera parameter set may be associated with multiple AR devices. 
     In some embodiments, setting the camera parameter in S 404  includes acquiring the camera parameter set from AR device that includes the camera through a data connection when the AR device becomes accessible by the one or more processors (S 410 ). For example, when the HMD  100  is connected (wired or wirelessly) to the AR device interface  309  of the computer  300 , the CPU  301  may retrieve the camera parameter set from HMD  100  (stored, for example, in the ROM  121 ). In other embodiments, the computer  300  may acquire the camera parameter set from the AR device by determining the camera parameter set. For example, the computer  300  may cause the camera  60  in the HMD  100  to capture one or more image frames of, for example, a calibration sheet and the computer  300  may analyze the resulting image frame (s) to determine the camera parameter set. In still other embodiments, the computer  300  may retrieve from the AR device an identification of the AR device and/or the camera in the AR device and retrieve the appropriate camera parameter set from the one or more memories based on the retrieved identification. As mentioned above, the various techniques may be combined. For example, in some embodiments, if the AR device is available to the computer (e.g., it is connected to AR device interface  309 ), the camera parameter set is acquired from the camera, and if the AR device is not available to the computer the setting of S 406  and S 408  is performed. 
     Once the camera parameter set is set, the CPU  301  generates at least one 2D synthetic image based on the camera parameter set by rendering the 3D model in a view range (S 414 ). The view range is the range of potential locations of the camera  60  around the stationary object for which images will be synthesized. In the example embodiment, the view range includes an azimuth component and an elevation component. The view range may also include a distance component that sets a distance of the potential locations in the view range from the 3D model of the object. The view range generally defines an area on the surface of a sphere having a radius equal to the length of the distance component. Each view point within the view range for which a synthetic image is generated represents a different pose of the object. 
     In some embodiments, the CPU  301  receives selection of data representing the view range (S 412 ) before generating the at least one 2D synthetic image. The selection may be received, for example, from a user selection via a GUI, such as the GUI shown and discussed later for  FIG. 5 . In some embodiments, the GUI includes a preview view of the object and a graphical representation of the user selected view range. In some embodiments, the view range is a single pose of the object selected by the user. In other embodiments, the view range is a predetermined (e.g., a default) view range. Instill other embodiments, the CPU  301  utilizes the predetermined view range unless the user provides a different selection of the view range (or modification of the predetermined view range. In some embodiments the predetermined view range is less than 360 degrees around the object in one or more of the azimuth or elevation. The view range will be explained in more detail below with reference to  FIGS. 5 and 6 . 
     The CPU  301  generates at least one 2D synthetic image of the 3D model representing the view of the 3D model from a location within the view range. The number of 2D synthetic images to be generated may be fixed, variable, or user selectable. Any suitable number of images may be generated as long as at least one 2D synthetic image is generated. If a single 2D synthetic image is generated, the image is generated for a central point within the view range. If more than one image is generated, the images are generated relatively evenly throughout the view range. In some embodiments, if the number of views is fixed or set by the user, the computer  300  determines how far apart within the view range to separate each image to achieve some distribution of images within the view range such as an even distribution (e.g., so that each image is a view from a same distance away from the view of each adjacent image). In other embodiments, the computer  300  generates a variable number of images, based on the size of the view range and a fixed interval for the images. For example, the computer may generate an image from a viewpoint every degree, every five degrees, every ten degrees, every twenty degrees in azimuth and elevation within the view range. The intervals above are examples and any other suitable interval, including less than a full degree interval, may be used. The interval between images does not need to be the same for azimuth and elevation. 
     The computer  300  generates the at least one 2D synthetic image based on the camera parameter set that was set in S 404 . The camera parameter set alters the rendering of the 3D object for the view point of the image to replicate a real image of the real-world object taken from the same viewpoint. In this embodiment, a process of generating synthetic images uses a rigid body transformation matrix for transforming 3D coordinate values of 3D points represented in the 3D model coordinate system to ones represented in an imaginary camera coordinate system, and a perspective projection transformation matrix for projecting the transformed 3D coordinate values to 2D coordinate values on the virtual plane of the synthetic images. The rigid body transformation matrix corresponds to a viewpoint, or simply a view, and is expressed by a rotation matrix representing rotations around three axes which are orthogonal to each other, and a translation vector representing translations along the three axes. The perspective projection transformation matrix includes camera parameters, and is appropriately adjusted so that the virtual plane corresponds to an imaging surface of a camera, such as camera  60 . The 3D model may be a CAD model. For each view, the computer  300  transforms and projects 3D points on the 3D model to 2D points on the virtual plane so that a synthetic image is generated, by applying rigid body transformation and perspective projection transformation to the 3D points. 
     In S 416 , the computer  300  generates training data using the at least one 2D synthetic image to train an object detection algorithm. The training data based on the synthetic image may be generated using any technique suitable for use with real images. In some embodiments, generating the training data includes generating an appearance template and/or a shape template using the 2D synthetic image (S 418 ). The appearance template includes one or more features such as color, surface images or text, corners, and the like. The appearance template may include, for example, (i) coordinate values of the locations of features of the object in the 2D synthetic image and their characterization, (ii) the coordinates of locations on the 3D model that correspond to those 2D locations and are represented in the 3D model frame of reference, and (iii) the pose at which the 3D model is rendered for the 2D image. The shape template describes the shape of the object in two dimensions without the surface features that are included in the appearance template. The shape template may include, for example, (i) coordinate values of points (2D contour points) included in a contour line (hereinafter, also simply referred to as a “contour”) representing an exterior of the object in the 2D synthetic image, (ii) the points on the 3D model that correspond to the 2D contour points and are represented in the 3D model frame of reference, and (iii) the pose at which the 3D model is rendered for the 2D image. In some embodiments, separate shape and appearance templates are created for each synthetic image generated for the view range. In other embodiments, data for multiple images may be stored in a single template. 
     The generated training data is stored in one or more memories (S 419 ). In some embodiments, the training data is stored in the computer&#39;s training system memory  305 . In some embodiments, when the HMD  100  is communicatively coupled to the computer  300  through the AR device interface  309 , the training data is stored by the computer  300  in the memory (such as ROM  121 ) of the HMD  100 . In other embodiments, the training data is stored in the computer&#39;s training system memory  305  and the HMD  100 . 
     After the training data is stored in the HMD  100 , the HMD  100  may operate to detect the object based on the training data. In some embodiments, the HMD attempts to detect the object in image frames of a real scene captured by the camera  60  by attempting to find a match between the template(s) and the image using the HMD&#39;s object detection algorithm. 
     In some embodiments, training data is generated for multiple cameras and/or AR devices for use in detecting a pose of the object in a real scene. In some such embodiments, setting the camera parameter set in S 404  includes setting a plurality of camera parameter sets for a plurality of cameras, S 414  includes generating a plurality of 2D synthetic images based at least on the plurality of camera parameter sets, and S 416  includes generating training data using the plurality of 2D synthetic images to train an object detection algorithm for a plurality of AR devices having the plurality of cameras. In other embodiments, steps S 404 , S 414 , and S 416  (optionally including one or more of S 406 , S 408 , S 410 , S 412 , and S 418 ) are simply repeated multiple times, each time for a different camera. 
     As mentioned above, in some embodiments, the computer performing the method  400  includes a GUI for providing information to and receiving selections from a user. FIGS.  5  and  6  are images of a GUI that may be used as part of a system that implements method  400 . For example, the GUI may be displayed by the computer  300  on the display unit  302  and be responsive to user input via the operation unit  304 .  FIG. 5  is a GUI  500  for receiving input from the user, and  FIG. 6  is a preview window  600  displaying a 3D model  602  of the object to assist in selection and/or preview of the view range. While the GUI shown in  FIGS. 5 and 6  will be described with reference to the method  400 , it should be understood that the method  400  may be performed without using the GUI of  FIGS. 5 and 6 , and the GUI shown in  FIGS. 5 and 6  may be used to perform training with methods other than the method  400 . 
     Turning initially to  FIG. 5 , the GUI  500  includes a model loading button  502  and a selected model display field  504  used to implement S 402  of the method  400 . In the example embodiment, when the user selects the model loading button  502 , another window is opened to allow the user to browse to the location (on the computer  300 , database or other storage location (whether local or remote)) at which a 3D model to be selected for training is located. The selected model display field  504  displays the location of the selected model, if a model has been selected. In some embodiments, the user may select the model by typing the location directly into the selected model display field  504 . Moreover, in some embodiments, the user may select the model by any other suitable technique, including by dragging and dropping the model into the GUI  500 . 
     The camera parameter set may be set in S 404  by selecting the AR device for which the object detection algorithm is to be trained in AR device selection field  506 . In the illustrative embodiment of  FIG. 5 , the AR device selection field  506  is a “pull down menu” or “drop down menu.” When the user selects the field  506 , a list of known AR devices is pulled down for selection by the user. In some embodiments, the AR device may be selected by detection by the computer  300  (for example as described in S 410 ). Based on the detected/selected AR device, the computer  300  retrieves the camera parameter set for the camera included in the AR device. The resolution and camera intrinsic parameters for the camera included in the AR device in field  506  are displayed in resolution field  508  and intrinsics field  510 . The resolution and camera intrinsic parameters collectively form the camera parameter set in this embodiment. 
     In some embodiments, the user may directly manually enter the camera parameters into the intrinsics field  510  and/or the resolution field  508  without having to complete step S 402  by specifying a number of already existing synthetic images. Also, the user may be allowed to modify the camera parameters that were inputted into the intrinsics field  510  and/or the resolution field  508  by the computer. Moreover, in some embodiments, the user may associate an entered/modified camera parameter set with a particular AR device or camera and store the new camera parameter set for future use. 
     A settings portion  512  of the GUI  500  allows the user to set and/or review the settings for generating synthetic images of the object  602  and training the object detection algorithm. The settings portion includes an azimuth setting section  514 , an elevation setting section  516 , a camera distance section  518 , a model rotation section  520 , a single view selector  524 , a lighting selector  526 , an internal rendering selector  528 , and a feature level selector  522 . 
     The settings portion  512  will be described with additional reference to  FIG. 6 . Model rotation section  520  allows the user to select the rotation of the model  602  of the object in three dimensions. In the example embodiment, the default rotation of the model is the rotation of the model  602  as stored in the model file. The model  602  is displayed in the preview window  600  according to the model rotation displayed in the model rotation section  520 . To change the rotation of the model, the user may numerically enter the rotation in the model rotation section  520  or rotate the model  602  in the preview window  600 , such as by selecting and dragging the model to rotate it (using a mouse, a user&#39;s finger on a touchscreen, or the like). The camera distance section  518  allows the user to set the distance from the model to the camera for which images will be synthesized. In some embodiments, the camera distance can also be modified by using the scroll wheel on a mouse to zoom in/out, or by any other suitable control. 
     The model  602  is displayed in the preview window  600  oriented according to the rotation selection and the camera distance selection, and it is partially surrounded by a mesh  604 . The mesh  604  defines a portion of a sphere around the model  602 , with the model  602  located at the center of the sphere. In some embodiments, the mesh  604  defines a full sphere around the model  602 . The mesh  604  is a visual aid to assist with visualizing the view range for creation for the synthetic images. The mesh includes elevation lines  608  and azimuth lines. Each azimuth line traces three hundred and sixty degrees in the azimuth direction  606  for a particular elevation. Similarly, each elevation line traces up to three hundred and sixty degrees in the elevation direction  608  for a given azimuth. In the illustrated example, the mesh  604  is not a full sphere and the elevation lines trace less than a full three hundred and sixty degrees. The azimuth and elevation lines define grid sections  610 . The grid sections  610  include selected grid sections  612  and unselected grid sections  614 . The selected grid sections  612  form the view range  616 . The view range  616  is the range of camera locations or orientations for which synthetic images are generated. The user may select the view range by selecting grid sections  610  in the preview window or by entering the view range numerically in the azimuth setting section  514 , the elevation setting section  516  of the window  500 . If the user enters the view range numerically, the appropriate grid sections  610  will be selected in the preview window  600 . Conversely, if the user selects (or unselects) grid sections  610  in the preview window  600 , the numerical representation in the input window  500  will be correspondingly updated. 
     In some embodiments, the view range  616  is preset. A default/preset view range may be fixed (i.e., not user changeable) in some embodiments. In other embodiments, the default/preset view range is changeable by the user. In an example embodiment, a default view range  616  is set with a range of sixty degrees of azimuth and forty five degrees of elevation. 
     Returning to  FIG. 5 , the single view selector  524  allows the user to select to synthesize a single view of the model  602 . If the single view selector  524  is not selected, multiple views of the model  602  are generated based on the selected view range. The lighting selector  526  allows the user to select whether or not lighting should be simulated when generating the synthetic images. The internal rendering selector  528  allows the user to select whether or not internal rendering should be used when generating the synthetic images. 
     The feature level selector  522  is used to indicate whether the model  602  is a rich feature object or a low feature object, which facilitates training the object detection algorithm. For example, the model  602  includes surface features  618  with distinct color, contrasts, shapes, etc. that may be used for object detection and tracking. Accordingly, rich feature is selected in the feature level selector  522  and the object detection algorithm will be trained to detect such features by, for example, creation of one or more appearance templates. If the model  602  did not include surface features  618 , or if the user did not want to use such features, low features would be selected in the feature level selector  522  and the object detection algorithm will be trained to detect the object without using surface features, such as by using a shape template. 
     When the user is ready to train the detection algorithm, the user selects the train button  532 . In response, the computer  300  generates the synthetic image(s) according to the settings the user selected (or the defaults), generates training data to train the object detection algorithm based on the synthetic image (s), and stores the training data. The stored training data is ready to be used by the AR device to detect and track the object and the user can proceed to train the algorithm for a different object or to train the algorithm to detect the same object using a different AR device. 
     The various embodiments described herein provide a system for auto-training an object detection algorithm using synthetic images. The embodiments reduce the amount of user involvement in training the algorithm, remove the time and effort needed to capture multiple images of an actual object using each particular AR device to be trained to detect the object, and remove the need to have an actual copy of the object and the AR device to be trained. 
     Some embodiments provide a non-transitory storage medium (e.g. ROM  121 , RAM  122 , identification target storage section  139 , etc.) containing program instructions that, when executed by a computer processor (e.g. CPU  140 , processor  167 , CPU  301 ), perform the methods described herein. 
     Coarse-To-Fine Template Matching 
     All template-based detection algorithms have the same tradeoff: speed vs. detection accuracy. The finer the 6DOF pose space is sampled during template matching, the better the accuracy at the expense of a linear increase in running time. A typical extension to standard template-based detection is to try to do the matching at a finer sampling level of 6DOF pose space without incurring a linear increase in running time. This is done by sampling the pose space in a hierarchical fashion. At the initial levels of the hierarchy, the search is performed at a coarse pose sampling. At later stages, the pose space around the candidates from previous levels is explored more finely. The refinement of sampling can be performed in all the six dimensions of pose. In embodiments where matching is performed in 2.5D space, the pose space is parameterized by (x, y, s, view), where x and y are image pixel position, s is the matching scale and view represents the view of the template using three angles. In one implementation, coarse-to-fine sampling is performed by refining the scale and the view in a hierarchical fashion. 
     For a finer scale matching, each matched view at scale s that passed the detection confidence threshold is also being matched at scale s±⅓Δs in a local neighborhood of 20×20 pixels, where Δs is the scale step size for template matching. The highest confidence detection out of the three (s−⅓Δs, s, s+⅓Δs) is chosen. 
     Various strategies can be implemented to perform finer matching of views. The main principle is to explore the views in a hierarchical manner, at each level one only has to match the candidates whose parents from the level above were detected. Therefore, parent views should be designed to guarantee that if a child view is detected, then the corresponding parent view should be detected as well. In one implementation, the parent view can be chosen to be one of the views in the cluster that is closest (in pose or in appearance) to all the other views in the cluster, which can be called a representative view. With that principle in mind, one is only left to define the view hierarchy. 
     According to one example, a 3-level hierarchy is chosen. The methods can distinguish between out-of-plane and in-plane rotation in the view hierarchy. At the top (1 st ) level, only representative out-of-plane rotations and representative in-plane rotations are explored. At the 2 nd  level, the system verifies the detected candidates with all in-plane rotations. At the last (3 rd ) level, the system verifies the detected candidates with all out-of-plane rotations. 
     To facilitate the construction of this hierarchy, one needs to decide how to cluster views at a fixed out-of-plane rotation based on in-plane rotation (moving from 1 st  to 2 nd  level) and how to cluster views based on out-of-plane rotation (moving from 2 nd  to 3 rd  level). In one implementation, when verifying views at level 2, the child views that are above the detection threshold are added to the list of candidates. When verifying views in level 3, only the best view survives. Either the best child view replaces the parent candidate or the parent candidate is kept if none of the children is better. 
     Considering a hierarchy for a simple 3D object, two different methods to cluster in-plane rotations and two different methods to cluster out-of-plane rotations are provided. In both cases, one method may be based on view proximity in 3D pose space and the other may be based on view appearance similarity. Below is a description of these methods. 
     Pose-Based View Clustering: 
     Out-Of-Plane Clustering 
     Views may be clustered based on their angular proximity on the view sphere  604 . Each view is represented as a point in 3D space, followed by K-means clustering. Currently the number of clusters is set to K=66. If the number of views on the view sphere  604  provided by the user is less than or equal to K, then no view clustering will be performed. For each cluster, the view that minimizes the sum of distances to all the other views within the cluster is selected as a representative. 
     In-Plane Clustering 
     When clustering all views at a fixed out-of-plane rotation based on pose, the method can assume that all the views are sampled at a fixed angular step and are ordered based on their in-plane rotation. Thus, in the current implementation, every K neighboring in-plane rotations can be grouped together and the middle template is chosen as a representative. 
     In the embodiment where K is 2, if the original sampling was every 9 degrees (40 equally spaced in-plane samples), then after clustering, views will be spaced 18 degrees apart. 
     Similarity-Based View Clustering 
     Simple man-made objects often exhibit a considerable amount of appearance similarity between different object views. This can be exploited in template matching. Instead of matching similar views using separate templates, a single template will be used first to detect one of these similar views and only then the individual templates will be used to determine the final view. Since no real image data is available at training time, to establish view similarity, template matching can be executed over computer generated edge images of all the template views. As a result, the system can obtain a N×N confusion matrix C, where C(i, j) is the matching score of template i to template j. Note that this is not a symmetric matrix. To convert it into a symmetric similarity matrix, similarity between templates i and j as S(i, j)=min(C(i, j), C(j, i)) can be determined. This similarity matrix S is used for both in-plane and out-of-plane view clustering. 
     Out-Of-Plane Clustering 
     Assuming the templates are ordered first by out-of-plane rotation and then by in-plane rotation, the N×N similarity matrix S is an M×M block matrix with P×P blocks, where N=M·P, M is the number of out-of-plane view samples, and P is the number of in-plane view samples. Each block corresponds to the similarity of all the templates of one particular out-of-plane rotation with all the templates of another out-of-plane rotation. View similarity S v (i, j) can be defined as the maximum similarity within each block corresponding to views i and j. Given this view similarity function, the system performs complete linkage hierarchical clustering over M object views. A stopping criteria may be defined when S v (i, j) goes below 50. This means that any two views within each cluster are guaranteed to have similarity of at least 50. For each cluster, the view that maximizes the sum of similarities to all other views within the cluster is selected as a representative. Note that each cluster can contain views that are far away in pose space if these views are similar enough in appearance, unlike in the case of pose-based clustering. 
     In-Plane Clustering 
     Once again, templates in one implementation can be ordered first by out-of-plane rotation and then by in-plane rotation. Then, the P×P blocks lying on the diagonal of matrix S are themselves similarity matrices S i  for views i=1 . . . M. Each P×P matrix S i  stores the similarities between all in-plane rotations of view i. Complete linkage hierarchical clustering can be used to group the in-plane rotations of each view i. Grouping currently stops when similarity goes below 70. For each cluster, the template that maximizes the sum of similarities to all other template within the cluster is selected as a representative. 
     Several clustering options have been presented herein. In general, more aggressive clustering to decrease the number of samples results in a faster run-time. On the other hand, accuracy may drop given more aggressive clustering. 
     Note that in cases of a highly symmetric object such as the gear a large speed up factor can be achieved, opening up the way for potentially exploring the view space more densely in the future. Also note that the speed-up factor is more significant for more cluttered scenes (see 1 and 5 objects vs. 10 objects case in the left subfigure). The evaluation was run on an Intel I5-2400 3.1 GHz CPU with 4 GB of RAM. The regions of interest were approximately 2 MP and 0.5 MP for the left and right tests, respectively. Note that only the effect on template matching time, not overall query time, has been measured. 
     Adaptive Thresholding 
     Current implementation of shape matching calculates the gradient orientation and their magnitude, and the orientation is kept only if the magnitude is higher than a Global Threshold set by the user. 
     This global threshold is a rather crude estimation of true edges, meaning some shadow edges can also have high gradient magnitude, and their orientation will be included in the matching phase. The effort of Adaptive Thresholding added here is to detect and remove shadow edges, by modeling true object edges as more uniformly illuminated within a local neighbourhood. One way to achieve this is by applying a box filter g[x,y] of a fixed size on an image I[x,y], and subtract with the original image. At any uniformly illuminated region, this difference D[x,y] will be very small:
 
 D [ x,y ]= I [ x,y ]−Σ g [ i,j ] I [ x−i,y−j ]
 
     A pixel is declared as an adaptive edge pixel if D[x, y] is larger than the preset adaptive threshold, for example, 0.01. 
     Depending on some situations, there may be two side effects of adaptive thresholding:
         a) Remove weak object edges—matching of shape templates depends on matching the sampled points on a specific view. The matching score can be affected by the missing points from a removed weak object edge.   b) Reduce the thickness of edges—makes it harder to match with a correct view based on the sampling frequency of object views.       

     A solution to these side effects is to take advantage of the 2 scales detection process of the shape matching algorithm. At the first and lower scale, step size is higher and image resolution is lower. Adaptive thresholding is applied at this level to remove very bad candidates caused by shadows. The correct views may not have the highest score, but they are still selected. At the second and higher scale, step size is smaller and image resolution is higher, normal thresholding is then used so that the matching score is not affected by removed points and thinner edges due to adaptive thresholding. Currently, this solution proves to maintain the correction detection rate at about the same level, but reduces the processing time, possibly because of the reduced time in processing shadow edges. 
     Adaptive Sampling 
     Consider a scenario where a dense set of real object images &amp; object&#39;s CAD model is available and a training tool needs to generate training data for low feature (LF) object detection and pose estimation (ODPE). The LF ODPE is dependent on the template matching of selected training images to the query scene. However, some objects have “unstable” poses which significantly change appearance if the object is slightly moved from the pose. Thus, if such a pose is selected for training, the detection of the object may be difficult when the user&#39;s viewpoint changes. This can result in loss of detection of the object if the viewpoint captures an unstable pose between two significantly different adjacent poses. Also, training view sampling algorithms may utilize uniform sampling resolution over a selected azimuth-elevation range. 
     One solution is to use “adaptive sampling” by varying the sampling density based on the stability of the object pose. For unstable poses, the sampling will be more dense such that the sudden variation of appearance will be well captured in the training set. 
     The sampling rate of training images on view sphere  604  can be determined by measuring the similarity of neighbor views in both “shape” and “appearance” space. 
     Regarding “shape” similarity, a process executed by CPU  301  may include generating edge maps densely from different view angles using a CAD model. Then, a “confusion” matrix can be constructed to measure shape similarity between different views. For each view, a similarity score or confusion score can be computed as compared with its neighbor views. The views with very low confusion score should be selected as unstable and hence include particularly distinctive poses for training. Shape template clustering can be performed in other view regions when less dense training is required. 
     Regarding “appearance” similarity, a process executed by CPU  301  may include comparing real training images or synthetic object images generated using a textured CAD model. A similarity between images from neighbor views can be measured using cross-correlation. Views having high correlation scores are considered to have similar appearance. They can be clustered and represented by fewer training images. For example, clustering may include deleting unneeded images, such as those that are very similar to their neighbors. The rest of the training images will be kept to describe unstable poses. 
     Since shape and appearance templates are used in different steps of LF detection pipeline, it is not necessary to require them to have the same sampling on the view sphere  604 . 
     Other aspects of the image data sets of adjacent images can be compared. In some embodiments, the feature data sets may be compared with respect to “intensity,” “color,” “ORB,” “edge,” etc. Other comparisons can be made by a hybrid process determined by machine learning. 
     For example, image features may refer to local attributes in the image data and/or 3D model data of a reference object. The image features may include edges/edge profiles, contours, corners, colors, textures, descriptor (e.g., including surface and direction information), etc., formed by such attributes, as well as more composite attributes that are learned by a machine from a collection of these local attributes. The learned processes may use sophisticated machine learning or deep feature learning methodology, which can distinguish the reference object among other surrounding objects in different environments, and approximately or accurately identify the view point or pose of the reference object. 
     Template clustering based on similarity is a common technique to build hierarchical template matching for better speed. Shape template clustering may be used in the implementation of shape matching. In some embodiments, clustering may be constrained in a predefined view range (e.g., 40 degrees×40 degrees) when the goal is to find unstable poses/views. The similarity computation and clustering may be performed in both shape and appearance space. 
     Adaptive sampling of training can help LF detection in two ways. First, adaptive sampling may improve the detection rate by preparing denser sampling in the view spaces that contain unstable poses/views to prevent a lower performance if the sampling remains sparse. Second, adaptive sampling may increase execution speed by using a minimal number of templates to represent stable poses/views. 
       FIG. 7  illustrates a flow diagram of a method  700  for performing adaptive sampling processes according to various embodiments of the present disclosure. The method  700  may include instructions stored on non-transitory computer-readable medium. In addition,  FIGS. 8A through 8D  show an example of some of the aspects of the method  700 . 
     The method  700  includes a step of performing a primary sampling process to create image data sets from primary sample views (“sample poses”), as indicated in block S 702 . The image data sets are created from a plurality of primary sample views of an object. The primary sample views, in some embodiments, may include points on a sphere or hemisphere surrounding the object.  FIG. 8A  shows locations  802  where the primary sample views may be obtained. Although shown in a square grid pattern, it should be noted that the locations  802  may include any suitable pattern. 
     The method  700  also includes a step (S 704 ) of storing the image data sets that are created from S 702 . With the stored image data sets, the data sets of adjacent or neighboring sampling points can be compared. Block S 706  indicates that a first primary view is selected, which may include, for example, primary view  804  shown in  FIG. 8B . From the perspective of this first primary view  804 , the method  700  includes selecting a first neighboring view, as indicated in block S 708 . For example, the neighboring views of primary view  804  may include at least views  806 ,  808 ,  810 , and  812  in  FIG. 8B . 
     Method  700  includes comparing the image data set of the selected primary view with the data set of the selected neighboring view to calculate a similarity score, as indicated in block S 710 . The similarity scores are indicative of the similarity between the images obtain from the neighboring or adjacent sample views. Thus, the method  700  is able to determine how much the images change when a small change in a viewing position is recorded. For example, the small change in viewing position can be defined by neighboring or adjacent sample views, such as the difference from a first sample view  610 ,  612  to another sample view  610 ,  612  offset from the first sample view  610 ,  612  in any direction along the sphere  604  in the azimuth direction  606  and/or elevation direction  608 . As shown in  FIG. 8B , respective similarity scores between view  804  and each of views  806 ,  808 ,  810 , and  812  are shown in FIG.  8 B, for reference purposes. 
     The method  700  further includes decision diamond S 712 , which indicates that it is determined whether or not the selected primary view has more neighboring views to analyze. If so, the method returns back to block S 708  so that the next neighboring view can be selected. If the primary view has no more neighboring views, the method  700  proceeds to the similarity score analyzing algorithm shown in  FIG. 7B . 
       FIG. 7B  shows a similarity score algorithm  718  of the method  700  for analyzing similarity scores between each of the primary views. Also, as will be discussed later, the similarity score algorithm  718  is also used to analyze similarity scores between other adjacent sampling points that are created between the primary sampling views at a finer resolution, or even points sampled at an even finer resolution (and repeated as many times as necessary to obtain favorable similarity scores). 
     The results of the comparison in S 710  can be stored in memory to be analyzed by a processor. As an example, the similarity score may range from 1 to 9, where 1 represents “no similarity” and 9 represents “exactly similar.” Other ranges may be used to represent varying degrees of similarity or “confusion.” Views having a high degree of similarity can also be said to be easily confused with each other. Similarity scores of 3, 7, 9, and 8 in the example of  FIG. 8B  are shown. 
     The method  700  further includes a step, as indicated by block S 720 , which includes analyzing a first similarity score obtained with respect to a first pair of adjacent sample views. According to decision diamond S 722 , the method  700  includes determining whether the similarity score exceeds a minimum threshold. The minimum threshold may be set at some intermediate value between the lowest and highest similarity scores. For example, when the similarity scores are calculated to span a range from 1 to 9, as mentioned in the example above, the minimum threshold score may be set somewhere between 2 and 8. For example, the minimum threshold may be 5. 
     According to some embodiments, the user may wish to create an overall sampling of the object that has a greater level of resolution. In this case, the system may allow the user to select any suitable level of resolution. In some embodiments, the user may utilize the image resolution setting  508  of the GUI  500  shown in  FIG. 5 . The method  700  may then, in some embodiments, adjust the minimum threshold accordingly. 
     For instance, to achieve a greater resolution level, the minimum threshold value regarding similarity scores would be raised (e.g., raised perhaps to 6, 7, or 8). Thus, with a higher minimum threshold, more iterations would be required to obtain more sample views since it would become more difficult to meet this higher similarity criteria, and thus more sample views would result in a higher resolution. Of course, more memory will be required to store the sample views in this high resolution training process. 
     On the other hand, the user may wish to reduce the amount of memory space used for storing the training data. In this case, the system may allow the user to select a lower resolution level, such as by setting the image resolution  508  to a lower level. Accordingly, the minimum threshold value analyzed in S 722  can be lowered (e.g., lowered perhaps to 2, 3, or 4). With a lower minimum threshold, fewer iterations would be required and fewer sample views would be obtained. 
     Returning again to the similarity score algorithm  718  of the method  700  of  FIG. 7B , if it is determined in S 722  that the similarity score exceeds the minimum threshold, no further samples are needed and the method  700  proceeds to decision diamond S 724 . For example, in  FIG. 8B , the similarity scores between views  804  and  806  is fine (greater than 5), between views  804  and  808  is fine, and between  804  and  810  is fine. As indicated in S 724 , the method  700  determines if more similarity scores are to be analyzed. If so, the method  700  returns to block S 720 , which indicates that the next similarity score is analyzed. 
     Otherwise, if the similarity scores between all pairs of adjacent sample views have been analyzed and no more scores are to be analyzed, the method  700  returns to decision diamond S 714  shown in  FIG. 7A . Decision diamond S 714  determines whether or not more primary views are to be analyzed. If so, the method  700  returns back to block S 706  so that the next primary view can be selected. If all the primary views have been analyzed, then the method  700  ends. 
     Returning back to  FIG. 7B , if it is determined in decision diamond S 722  that the analyzed similarity score does not exceed the minimum threshold (such as between views  804  and  812 , where the score is 3), then the method  700  proceeds to block S 726 , which indicates that an algorithm for performing a finer sampling resolution is performed. One finer sampling resolution algorithm is described with respect to  FIG. 7C . Regarding the finer sampling resolution algorithm S 726 , further processing will be needed if it is determined that the adjacent sample views are too dissimilar and more training samples would be needed in between the adjacent sample views. 
     For example, after the samples are obtained and the object is detected, the lack of necessary similar adjacent views may cause the object to be lost. If adjacent views do not meet the necessary similarity test, then the system may not be able to detect the object if the object is viewed from a location between the two adjacent views. Hence the object may be lost during use of the object detection system (i.e., after the training samples have already been obtained). 
       FIG. 7C  shows an example of the finer sampling algorithm  726 , which is similar to the creation of primary view sampling shown in  FIG. 7A . The finer sampling algorithm  726  includes a step (S 728 ) of selecting the “low score” neighboring view (e.g., view  812 ) that shares the low similarity score with the primary view  804 . Next, the method  700  includes a step of performing a finer sampling process to create additional image data sets from one or more finer sample views (“sample poses”), as indicated in block S 730 . The finer sample views, in some embodiments, may include points on the sphere or hemisphere surrounding the object in between the primary views.  FIG. 8C  shows an example of additional sample locations  820 ,  822 ,  824 ,  826 ,  828 , and  830  where the finer resolution samples of image data sets are obtained around “low score” sample  812 . 
     The algorithm  726  of method  700  also includes a step (S 732 ) of storing the additional image data sets. With the stored image data sets, the data sets of adjacent or neighboring sampling points can be compared. From the perspective of the suspect view, the method  700  includes selecting a first neighboring view that neighbors the low score view  812 , as indicated in block S 736 . 
     Similar to block S 710  of  FIG. 7A , method  700  includes block S 738 , which includes comparing the data set of the view  812  with the data set of the selected finer neighboring view to calculate a similarity score. Again, the similarity scores are indicative of the similarity between the images obtain from the neighboring or adjacent sample views. Thus, scores are calculated between view  812  and each of views  820 ,  822 ,  824 ,  826 ,  828 , and  830 . 
     The method  700  further includes decision diamond S 740 , which indicates that it is determined whether or not the view  812  has more finer neighboring views to analyze. If so, the method returns back to block S 736  so that the next neighboring view can be selected. If the suspect view has no more finer neighboring views, the method  700  proceeds to the similarity score analyzing algorithm  718  of  FIG. 7B . 
     It should be noted that the similarity scores are analyzed again in S 722  shown in  FIG. 7B . If the scores are low, such as is shown in  FIG. 8C  where the score between views  812  and  824  is only  2 , then the algorithm of  FIG. 7C  is executed again. For example, as shown in  FIG. 8D , the low score neighboring view  824  is further processed by performing even finer sampling (S 730 ) between view  824  and newly created even finer neighboring views  840 ,  842 ,  844 ,  846 ,  848 , and  850 . In this example, the similarity scores exceed the minimum threshold, so that a finer iteration is not needed. It should be noted that one iteration ( FIG. 8B ) may be needed, two iterations ( FIG. 8C ), three iterations ( FIG. 8D ), or more (not shown). 
       FIGS. 9A through 9C  show another example of a portion  900  of a hemisphere (or sphere  604 ) that surrounds an object from which training samples are obtained, according to another embodiment. The portion  900  of the hemisphere, in this example, includes sample view locations  902 . For example, the locations  902  may be located within window  616  shown in  FIG. 6 . During a first iteration ( FIG. 9B ), sample views are obtain at each of the locations  902 - 918 . 
     Although the locations  902 - 918  are shown in  FIGS. 9A-9C  in a rectangular pattern, it should be noted that the pattern of image sampling locations can include any configuration. 
       FIG. 9A  shows an example of a grid  800  of primary sample locations  802 ,  804 ,  806 ,  808 ,  810 ,  812 ,  814 ,  816 ,  818  where the primary samples are obtained. The locations may represent points on a sphere surrounding an object to be sampled. Although  FIG. 8A  shows the grid  800  with the locations  802  arranged in a square pattern, it should be noted that the locations may be arranged in any suitable pattern along any spherical or other shape around the object. In some embodiments, each location may have four, five, six, or any other number of neighboring locations that are positioned at approximately equal distances from one another. 
     As mentioned above with respect to block S 704  of the method  700  of  FIG. 7 , the neighboring sample views are compared to obtain primary sample views. For example, the sample view of primary location  910  may be compared with the four nearest neighbors  904 ,  908 ,  912 , and  916 . Similarity scores (SS) are calculated based on the similarity of the images of the sample views of the adjacent views. In this example, the SS values represent numbers from 1 to 9 defining how closely one adjacent view compares with another. The similarity between the views at locations  910  and  904  includes a score of 7 (i.e., SS=7). Between locations  910  and  908 , SS=7; between locations  910  and  912 , SS=4; and between locations  910  and  916 , SS=4. 
     Using a “minimum threshold” of 5, as described above with respect to decision diamond S 708  shown in  FIG. 7 , most of the primary sample views shown in  FIG. 9A  meet or exceed the minimum threshold. However, between locations  910  and  916 , the similarity does not meet or exceed the minimum (5), but SS=4. Also, between locations  910  and  912 , the similarity does not meet or exceed the minimum threshold. Therefore, for these two cases, according to this example, additional sampling is needed. 
       FIG. 9B  shows a closer view of the portion  900  of the hemisphere as compared with  FIG. 9A . In this expanded view, the relevant sampling locations  910 ,  912 , and  916  are shown in  FIG. 9B . Since additional sampling is needed between or near the pairs ( 910 / 912  and  910 / 916 ) of primary sampling locations because of the low similarity scores, secondary sampling locations  922 ,  924 ,  926 ,  928 , and  930  may be used. 
     In other embodiment, fewer or more secondary locations may be analyzed and the specific positioning can include any suitable arrangement such that the positions are in between and/or near the pairs of sampling locations where the similarity scores are low. For illustration purposes only, the second locations are shown as triangles, whereas the primary sampling locations are shown as circles. 
     In accordance with block S 712  of the method  700  of  FIG. 7 , secondary sampling is performed between the newly created secondary sampling locations and their neighboring locations. These new secondary sample locations  922 ,  924 ,  926 ,  928 , and  930  are compared, not only with each other, but also with other primary locations  910 ,  912 , and  916  that are neighboring.  FIG. 9B  also shows the similarity score (SS) values calculated between the neighboring views. In this example, all but two of the SSs meet or exceed the minimum threshold (e.g., 5), including the comparisons between secondary location  924  and secondary location  926  (i.e., SS=3) and between secondary location  928  and primary location  916  (i.e., SS=4). 
     The method  700  can be repeated any number of levels to obtain primary, secondary, tertiary, and more sample views as needed. In  FIG. 9C , a third round of sample locations are created to account for the low SS values shown in  FIG. 9B . In particular, the low SS values were calculated between locations  924  and  926 , and between  928  and  916 . These locations are shown in  FIG. 9C , which shows an even greater expanded view of portion  900  of the hemisphere. Additional sample locations  934 ,  936 ,  938 ,  940 ,  942 , and  944  (depicted as squares) are created. These locations  934 - 944  are created between and/or near the pairs of neighboring locations where SS values are low. In the example of  FIG. 9C , the third iteration is enough to fine tune the samples to fill in the areas where sample images experience a more dramatic change over a short distance. The SS values reflect the fact that the nearer locations in the tertiary iteration provide suitable similarities between images. 
       FIG. 10  shows an example of an object that has been sampled at ten different positions. As can be seen from a comparison of adjacent images, the changes may be slight, perhaps slight enough where the SS values calculated between the neighboring images would not require secondary sampling. However, if one or more additional iterations are needed to obtain samples that have SS values that exceed the minimum threshold, then the method  700  or other processes can be performed to achieve adequate training samples. 
     Although the invention has been described with reference to embodiments herein, those embodiments do not limit the scope of the invention. Modifications to those embodiments or different embodiments may fall within the scope of the invention.