Abstract:
In a method and system of training, one or more digital assets, such as two-dimensional and three-dimensional computer-generated objects, are superimposed over a live camera view to generate a simulated training scenario, referred to herein as augmented reality (“AR”) technology. By leveraging AR technology, a live simulation of real-world events, situations, and skills is generated for which an employee, student, customer, or any type of person in need of training, is being trained. A trainee is thus immersed directly into the training material. The physical environment and working conditions are integrated into the sequence of training material, and it does this live, that is, with immediate feedback from a camera&#39;s live screen view. This technique may, by way of examples, also be described as an automated guided tour used to facilitate employee on-boarding and training, or as a guided tour through a warehouse store for its customers.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims the benefit of U.S. Provisional Application No. 62/130,391, filed Mar. 9, 2015, which application is hereby incorporated herein by reference, in its entirety. 
     
    
     TECHNICAL FIELD 
       [0002]    The invention relates generally to training and, more particularly, to using augmented reality to enhance the impact and effectiveness of training of persons, such as employees, customers, students, and any person or persons in need of training. 
       BACKGROUND 
       [0003]    Retailers and manufacturers are two sectors of the economy that rely on acquiring and retaining a substantial number of unskilled employees required to perform various operations. One of the key challenges in bringing on such employees is to provide them with the knowledge and understanding required to perform their new jobs safely and effectively. One of the key strategies used to provide this knowledge is through the use of training tools such as written training material, classroom group-based-training, one-on-one training with a supervisor, and on-line training using text, video, photography, and electronic quizzes. Each of these techniques suffers from limitations of effectiveness, such as (1) high cost (including direct cost and/or time spent away from one&#39;s job duties), (2) low relevance (e.g., images or videos that are not specific to a respective industry or environment, or that do not, for example, accurately and/or realistically portray dangerous situations), and (3) low impact (e.g., information is not retained for much time after delivery to an employee). These limitations also apply to other types of trainees such as customers shopping in warehouse stores, students enrolled in technical courses (i.e., welding, electrical), or any person in need of training. 
         [0004]    Therefore, what is needed is a system and method for training people, such as employees, customers, and students, that is cost effective, relevant, and that will have a lasting impact on an employee&#39;s performance. 
       SUMMARY 
       [0005]    The present invention, accordingly, solves these problems in a unique and compelling manner by superimposing one or more digital assets, such as two-dimensional and three-dimensional computer-generated objects, over a live camera view to generate a simulated training scenario, referred to herein as augmented reality (“AR”). By leveraging AR technology, a live simulation is generated of real-world events, situations, and skills for which a person is being trained. In effect, a person is immersed directly into the training material. The physical environment and working conditions are integrated into the sequence of training material, and it does this live, that is, with immediate feedback from a camera&#39;s live screen view. This technique may also be described as an automated guided tour used to facilitate employee on-boarding and student training. 
         [0006]    The foregoing has outlined rather broadly the features and technical advantages of the present invention in order that the detailed description of the invention that follows may be better understood. Additional features and advantages of the invention will be described hereinafter which form the subject of the claims of the invention. It should be appreciated by those skilled in the art that the conception and the specific embodiment disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: 
           [0008]      FIG. 1  exemplifies a schematic overview of software architecture and features of the present invention; 
           [0009]      FIG. 2  exemplifies a training session of the system of  FIG. 1 , the session having a number of lessons; 
           [0010]      FIG. 3  exemplifies a camera view of a juice storage bin of  FIG. 1 ; 
           [0011]      FIG. 4  depicts a live camera view of  FIG. 3 ; 
           [0012]      FIG. 5  exemplifies a trigger image of  FIGS. 3 and 4 ; 
           [0013]      FIG. 6  exemplifies an augmented camera view of the live camera view of  FIG. 4 ; 
           [0014]      FIG. 7  shows a live camera view exemplified in an additional application of the invention; 
           [0015]      FIG. 8  exemplifies an augmented camera view of the live camera view of  FIG. 8 ; 
           [0016]      FIG. 9  exemplifies a quiz subsequent to the lesson of  FIGS. 3-8 ; 
           [0017]      FIG. 10  depicts a flow chart of steps executable in accordance with the invention for training; and 
           [0018]      FIG. 11  depicts a flow chart of steps executable in accordance with the invention for displaying a training situation. 
       
    
    
     DETAILED DESCRIPTION 
       [0019]    The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. Additionally, as used herein, the term “substantially” is to be construed as a term of approximation. 
         [0020]    It is noted that, unless indicated otherwise, functions described herein may be performed by a processor such as a microprocessor, a controller, a microcontroller, an application-specific integrated circuit (ASIC), an electronic data processor, a computer, or the like, in accordance with code, such as program code, software, integrated circuits, and/or the like that are coded to perform such functions. Furthermore, it is considered that the design, development, and implementation details of all such code would be apparent to a person having ordinary skill in the art based upon a review of the present description of the invention. Such a person having ordinary skill in the art would be able to make use of commercially-available software tools, components, and libraries to build a software application that implements the system being described. 
         [0021]    Referring to  FIG. 1  of the drawings, the reference numeral  100  generally designates a mobile computing device, represented herein as a tablet, configured to embody features of the present invention. The tablet  100  includes a central processing unit (also referred to herein as a “CPU” or “processor”)  101  coupled to a memory  102  having an application software program  103  executable by processor  101  for training persons, as described in further detail below. A display  108  is coupled via a graphics rendering engine  110  to CPU  101 . Display  108  may include a display built into tablet  100  or alternative displays, such as an optical head-mounted display (OHMD, e.g., Google Glass) or the like. One or more speakers  112  are preferably coupled via audio hardware  114  to CPU  101 . 
         [0022]    Tablet  100  includes position sensors  116 , such as gyroscopes, which are effective for generating data indicative of the location and orientation of tablet  100  relative to the target image, or object,  128 . Position sensors  116  are coupled (through CPU  101 ) to memory  102  for inputting the position data to tablet location orientation software module  118  which is run by CPU  101  through application software program  103  for determining the location and orientation of tablet  100  relative to the target image, or object,  128 , and saving that information into the memory  102 . A camera  120  is mounted on tablet  100  for capturing a camera view  121 , preferably live, of an object  128 , exemplified as a juice storage bin having a trigger image  130  and the object&#39;s environment, and for generating image data indicative of camera view  121  captured by the camera. Camera  120  is coupled (through CPU  101 ) to memory  102  for inputting the image data to an image recognition software engine  122  which generates an image signal to an image location orientation software module  124 , which is run by CPU  101  through application software program  103  for determining the position and orientation of object  128  of the image and saving that information into the memory  102 . By way of example, with location and orientation of the tablet and target image determined, if the tablet is three feet away from the target image  128 , then the 3D object (augmentation) is rendered a certain size. If the tablet is further from the target image, then the 3D object is rendered smaller in size, just like a real object. With respect to orientation, if the tablet is above the target image, the augmentation is rendered as if looking down at the target image from above. If the tablet is looking at the target image from the side, then the augmentation is rendered as if looking at that side of the target image. It is understood that  FIG. 1  is a schematic drawing and, as such, camera  120  and tablet position sensors  116  are actually coupled to memory  102  through the CPU  101 . 
         [0023]    As further shown in  FIG. 1 , in memory  102 , application software program  103  is operative with AR sequence definitions  106 , which are training programs designed for a specific purpose or job role. By way of example, but not limitation, AR sequence definitions may be designed for “New Employee Training at Retail Grocery” or “Poultry Processing Employee Training.” Each AR sequence definition  106  comprises a number of sequences, also known as lessons, such as “How to properly clean a deli slicer” or “How (and why) to clean up a water spill”). Each sequence, or lesson, comprises one or more steps, each of which steps comprises one or more digital media, or assets,  104 . 
         [0024]    Digital assets  104  include one or more of 2D and 3D objects, audio clips (e.g., of human voice instructions or procedures, sample sounds of machinery or devices, music, and the like), video clips (e.g., of instructions, procedures, dramatizations of problems or incidents, corporate messaging, and the like), static images (e.g., of technical drawings, depictions or photographs of machinery, equipment, illustrations, photographs of problems of incidents, and the like), text data (e.g., of instructions, procedures, statistics, and the like), animations (e.g., of instructions, procedures, and the like), hyperlinks (e.g., to documentation, reports, external applications, and the like), and any other types of digital media. 
         [0025]      FIG. 2  of the drawings exemplifies an overview of the system of  FIG. 1  for executing a training session having a number of lessons. The system is configured for training an employee  126 , also referred to herein as a trainee, having tablet  100  configured to instruct the trainee to start the training application and follow the instructions that appear on the screen of the tablet. This is one specific example of the application of this technology to provide an effective on-boarding program to new employees of a retail grocery store. 
         [0026]    As shown in  FIG. 2 , the application preferably instructs employee  126  to go to a specific location in the work environment. The location could be a site of a particular machine, piece of equipment, or component of a process. The employee walks to that particular location in the work environment. The employee is preferably not accompanied by a manager, supervisor, or any other employee. The employee preferably proceeds through the lesson in a self-guided manner. As exemplified in  FIG. 2 , and discussed in further detail below with respect to  FIGS. 3-9 , the exemplified training session includes a number of lessons, such as how to respond to a water spill (lesson  1 ), how to handle bacterial growth on a deli slicer  134  (lesson  2 ), and the like. 
         [0027]    Referring to  FIGS. 3-5 , as employee  126  reaches the required location for training, tablet  100  training application software program  103  preferably displays on display  108  a live camera view  140  ( FIG. 4 ) and instructs employee  126  to look for a trigger image  130  ( FIG. 5 ) having a particular pre-defined trigger image tag  132  that is attached to some part of the equipment. Image tag  132  is preferably a barcode, a QR code, a customer&#39;s store logo, or any kind of uniquely identifiable image that may reside on a sticker or label visibly applied to equipment that is the object of training. 
         [0028]    Once the training application software program  103  detects image tag  132  in the camera&#39;s live view  140 , the training application program preferably generates an AR overlay  142 , using 3D renderings  144  and  146  selected from digital assets  104 , on top of the camera view  121  ( FIG. 6 ) that the trainee views on display  108 . The AR overlay  142  preferably comprises digital assets  104 , including images, text, video, 3D renderings, sound effects, vibration, animated 3D renderings, and/or the like. By using the AR overlay elements, digital assets  104 , the training application software program  103  provides a training sequence, or lesson, that incorporates elements of the live camera view  140  of the trainee&#39;s environment to teach the subject matter of the lesson. It may be appreciated that the invention can provide a simulation and impact of an unsafe, threatening, and/or costly situation, or of a complex, multi-step procedure, without the cost or actual exposure to the employee of those actual conditions. By way of examples, but not limitations, such training sequences, or lessons may include: 
         [0029]    1. A water spill on the floor next to some type of cooling equipment. This is exemplified in  FIGS. 1-5 , wherein a water-monster tentacle  146  ( FIG. 6 ) is animated reaching up through a puddle  144  to threaten an employee. In the case of the water spill, the system can render multiple tools, as exemplified in  FIG. 9 , that could be used to clean up the spill, such as a mop, paper towels, squeegee, etc. These tools are preferably rendered in proportionate scale and proper orientation relative to actual objects in the physical environment, enhancing the realism of the training exercise; 
         [0030]    2. A contaminated piece of food cutting equipment. By way of example, a deli slicer  134 , shown as clean in  FIGS. 2 and 7 , is rendered in  FIG. 8  with bacteria on the surface of the cutting blade  164  greatly magnified to make the bacteria visible to the eye. 
         [0031]    3. A theft in progress in a retail environment can be simulated much more cost-effectively than a training exercise setup that involves other employees or paid actors. 
         [0032]    4. A fire in a chemical storage cabinet. 
         [0033]    5. An event external to the work environment, such as a dangerous weather event or loss of power. 
         [0034]    6. A multi-step procedure to safely disassemble, clean, and reassemble a commercial-quality electric deli meat slicer. 
         [0035]    7. Spills of poisonous liquids. 
         [0036]    8. Broken glass on a floor. 
         [0037]    9. Contaminated food. 
         [0038]    10. Confined spaces. 
         [0039]    11. Equipment accidents and failures. 
         [0040]    12. A simulation, or 3D rendering, of equipment to train on that is not present in the trainee&#39;s work environment for various reasons, such as the equipment being only occasionally available or usable, out for repairs, the equipment being rental equipment, or the like. 
         [0041]    By integrating sounds, sights, and conditions of the actual work environment into the simulation, the simulation becomes as realistic as possible, and is therefore much more effective in achieving suspension of disbelief. It is more convincing to the trainee that they are actually seeing the event happen in front of them and that they are actively participating in it, instead of merely watching it passively from a distance and at a different time. In the above example of a theft in progress in a retail environment, the trainee sees for him or herself the complexity of a theft situation when there are other employees and even customers in the same environment that could be affected by the situation at hand. In such a case, the trainee is much more likely to be emotionally invested in the situation, and to be deeply motivated to pay attention to the lesson being taught and especially to the remedies and solutions that the training provides. 
         [0042]    Depending on the specific lesson, the AR digital assets  104  can be used to simulate the tools and techniques required to help remedy the situation in the lesson being taught. 
         [0043]    Operation of the system is exemplified by flow charts  200  and  300 , depicted in  FIGS. 10 and 11 , respectively. Accordingly, in step  202  of  FIG. 10 , a trainee is presented with a set of instructions about what they should do and what tools they should use to remedy the situation, as discussed in further detail in the following, with respect to  FIG. 11 . 
         [0044]    Referring to  FIG. 11 , in operation, at steps  302  and  304 , camera view  121  is passed to the image recognition engine  122  which recognizes object  128  and detects a trigger image  130 , determines the trigger image&#39;s location and orientation, and passes that information to application software program  103 . At substantially the same time, at step  306 , the tablet&#39;s position sensors  116  and location and orientation module  118  provide the location and orientation of the tablet and target image  128 , and passes that information to program  103 . When both the trigger image&#39;s location and orientation as well as the tablet&#39;s location and orientation are known, at step  308 , execution by program  103  of the AR sequence definition  106  begins. 
         [0045]    Execution of the AR sequence definition, or lesson,  106  comprises a series of one or more steps in the AR training course, each of which steps calls up one or more digital assets  104 . Execution of the first step or next step in the series of steps begins at step  310 . As shown at step  312 , each step of the execution generates a composite visual output comprising the original camera view  121 , with one or more of the above digital assets  104  superimposed over the camera view and, at step  314 , the 2D and 3D objects are adjusted in location and orientation such that their apparent position orientation remains fixed with respect to the physical objects in the scene. At step  316 , the composite view is passed on to the graphics rendering engine  118  of the tablet and is output to the tablet&#39;s visual display  108 . At step  318 , audio assets  104  are sent to the audio hardware  114  to be played on the tablet&#39;s speakers  112  in coordinated sequence with the visual objects. At step  320 , a determination is made whether there are more steps in the lesson. If there are more steps, execution returns to step  308  to thereby guide the trainee from one step to the next, in exactly the correct order as required by the procedure and as built into the application software program  103  and AR sequence definition  106 . If there are no further steps for this lesson, execution proceeds to step  204  of  FIG. 10 . 
         [0046]    At step  204 , the trainee is given a quiz (e.g.,  FIG. 9 ) and prompted to answer one or more questions that test their understanding of the material of the lesson, based on the situation that is being simulated. Such questions may include 3D-rendered objects that simulate options available to address a problem of a given situation. Questions may even include multiple correct choices, each of which may reduce or eliminate the problem, and not necessarily in the same way as other correct options. 
         [0047]    At step  206 , if the trainee chooses an incorrect option, execution proceeds to step  208  wherein the AR digital assets  104  are preferably used to simulate the effects of the incorrect choice. The effects could be that the problem does not get remedied, or the problem gets even worse, or a new and possibly more dangerous problem is created. 3D animated renderings and audio effects may be generated to simulate the result of an incorrect choice in a multiple-choice question, and illustrate to the trainee how an incorrect choice either has no effect on a problem, or causes the problem to get worse, or causes another, potentially worse problem to arise. 
         [0048]    At step  210 , the application returns to step  202  and enters a remediation loop to re-present the material, possibly in an alternative format, order, or style, preferably adapting to the user&#39;s personal learning style and preferences, to reinforce the required lesson. 
         [0049]    If at step  206 , a trainee correctly answers the one or more questions, that lesson is counted as complete and, optionally, 3D animated renderings and audio effects are displayed to simulate the result of one or more correct responses and illustrate to the trainee how one or more correct responses cause the problem to be solved or ill effects to be minimized. A determination is then made at step  212  whether there are more lessons for the topic at hand. If it is determined that there are more lessons, execution proceeds to step  214 , wherein the application instructs the trainee to move to another location in the work environment where the next lesson will be displayed. For example, with reference to  FIG. 2 , a trainee may move from lesson  1  to lesson  2 . Execution then returns to step  202 . The trainee thus moves through a series of lessons as described above that comprise the set of lessons required for a specific topic. 
         [0050]    If, at step  212 , it is determined that there are no more lessons to cover, then at step  216 , the application software program  103  will store as many lessons as necessary for a specific topic. After visiting all of the image tags and completing the sequence of lessons, the trainee completes the training session. 
         [0051]    At step  218 , the results of the training are uploaded to a cloud-based service and stored. The results are then analyzed for particular weaknesses in the trainee&#39;s results. A report is then generated for the trainee&#39;s supervisor. The report will direct a supervisor to provide, if necessary, further remedial training by means of a talk for the specific areas of weakness. No remedial actions are generated or suggested for topics for which the trainee exhibited satisfactory results. 
         [0052]    Optionally, upon completion of the entire sequence of steps, the final result of the procedure may be simulated as a set of 3D renderings and audio effects and the like superimposed on the live view, the audio effects being played through the speakers incorporated into the mobile computing device, thereby providing positive feedback to the trainee. 
         [0053]    It is understood that the present invention may take many forms and embodiments. Accordingly, several variations may be made in the foregoing without departing from the spirit or the scope of the invention. For example, the training system and method may be configured to adapt to incorrect choices and an employee&#39;s learning style and preferences. In another example, instructions and orientation may be auditory, that is, verbal. In still another example, the system and method may be used to screen or weed out high-risk candidates (e.g., based on a report of results in step  218 ) before they are hired. In yet another example, the application may be completely self-contained, without any capability to upload data to a cloud-based or server-based central application, but instead contains within itself additional functions to generate reports and summaries for supervisor or instructor review. 
         [0054]    By use of the present invention, an automated training system provides on-boarding and continuous learning, using a built-as-a-mobile-application which a trainee can use and follow completely on their own with little or no supervision or guidance. The relevance, effectiveness, and impact of the teaching material is enhanced, while training costs are reduced. 
         [0055]    Further, training lessons are made more realistic by incorporating elements of the actual physical environment into the lesson, wherein elements include at least one of objects or equipment, employees, customers, lighting levels, noise levels, smell, vibration, and temperature, and the like. 
         [0056]    Still further, the 3D renderings and audio effects generate an exaggerated perspective of a situation that makes the lesson more believable as an actual live event, instead of a recording or simulation of an event, and thereby make the training more memorable. 
         [0057]    Still further, multiple mobile devices may be configured to enable a single supervisor, instructor, or customer representative to provide multiple training sessions in parallel. 
         [0058]    Still further, a user could be prompted with 3D animated renderings to choose among a series of options, including at least one of a multiple-choice question that represents real-world strategies to resolve a situation presented in a lesson. By way of example, a water spill scenario could show three options: a mop, a broom, and paper towels. Instead of rendering these options as merely text or images, they could be rendered as 3D objects. When selected, the objects would be animated to show the result of the use of that tool. Selecting the broom would show the broom moving the water back and forth, but no progress made in collecting the water. Selecting the mop would show the mop circling the spill and absorbing the liquid. Having thus described the present invention by reference to certain of its preferred embodiments, it is noted that the embodiments disclosed are illustrative rather than limiting in nature and that a wide range of variations, modifications, changes, and substitutions are contemplated in the foregoing disclosure and, in some instances, some features of the present invention may be employed without a corresponding use of the other features. Many such variations and modifications may be considered obvious and desirable by those skilled in the art based upon a review of the foregoing description of preferred embodiments. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the invention.