Patent Publication Number: US-2023141037-A1

Title: System and method for providing weakly-supervised online action segmentation

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims priority to U.S. Provisional Application Ser. No. 63/278,001 filed on Nov. 10, 2021, which is expressly incorporated herein by reference. 
    
    
     BACKGROUND 
     Human action understanding from video, which involves recognizing, localizing, and forecasting human behavior is an important and widely studied problem in the quest for visual intelligence. Action understanding in videos is important in many applications where agents learn by observation of other agents performing complex tasks that often involve interactions with objects. Creating fully annotated clips with action assignments and labels on the temporal boundaries of individual actions is manually intensive and is therefore both time consuming and expensive. This limits the scale and practicality at which fully supervised video datasets may be created. Furthermore, the subjective nature of labeling start and end times of each action results in ambiguities for evaluation. 
     BRIEF DESCRIPTION 
     According to one aspect, a computer-implemented method for providing weakly-supervised online action segmentation that includes receiving image data associated with multi-view videos of a procedure. The procedure involves a plurality of atomic actions. The computer-implemented method also includes analyzing the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels. The computer-implemented method additionally includes training a neural network with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation. The computer-implemented method further includes executing online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network. At least one computing system is controlled to provide automation or feedback with respect to real-time atomic actions involved in completing the procedure based on the online action segmentation. 
     According to another aspect, a system for providing weakly-supervised online action segmentation that includes a memory storing instructions when executed by a processor cause the processor to receive image data associated with multi-view videos of a procedure. The procedure involves a plurality of atomic actions. The instructions also cause the processor to analyze the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels. The instructions additionally cause the processor to train a neural network with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation. The instructions further cause the processor to execute online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network. At least one computing system is controlled to provide automation or feedback with respect to real-time atomic actions involved in completing the procedure based on the online action segmentation. 
     According to yet another aspect, a non-transitory computer readable storage medium storing instruction that when executed by a computer, which includes a processor perform a method that includes receiving image data associated with multi-view videos of a procedure. The procedure involves a plurality of atomic actions. The method also includes analyzing the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels. The method additionally includes training a neural network with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation. The method further includes executing online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network. At least one computing system is controlled to provide automation or feedback with respect to real-time atomic actions involved in completing the procedure based on the online action segmentation. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The novel features believed to be characteristic of the disclosure are set forth in the appended claims. In the descriptions that follow, like parts are marked throughout the specification and drawings with the same numerals, respectively. The drawing figures are not necessarily drawn to scale and certain figures can be shown in exaggerated or generalized form in the interest of clarity and conciseness. The disclosure itself, however, as well as a preferred mode of use, further objects and advances thereof, will be best understood by reference to the following detailed description of illustrative embodiments when read in conjunction with the accompanying drawings, wherein: 
         FIG.  1    is a schematic view of an exemplary operating environment for providing weakly-supervised online action segmentation according to an exemplary embodiment of the present disclosure; 
         FIG.  2    is a schematic overview of an online action segmentation application according to an exemplary embodiment of the present disclosure; 
         FIG.  3    is a process flow diagram of a method for training a neural network with respect to a plurality of atomic actions of a procedure within a training mode of the online action segmentation application according to an exemplary embodiment of the present disclosure; 
         FIG.  4    is a process flow diagram of a method for completing online action segmentation to label atomic actions that are occurring in real-time according to an exemplary embodiment of the present disclosure; and 
         FIG.  5    is a process flow diagram of a method for providing weakly-supervised online action segmentation according to an exemplary embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that may be used for implementation. The examples are not intended to be limiting. 
     A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus may transfer data between the computer components. The bus may be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus can also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others. 
     “Computer communication”, as used herein, refers to a communication between two or more computing devices (e.g., computer, personal digital assistant, cellular telephone, network device) and can be, for example, a network transfer, a file transfer, an applet transfer, an email, a hypertext transfer protocol (HTTP) transfer, and so on. A computer communication can occur across, for example, a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a circuit switching system, a packet switching system, among others. 
     A “disk”, as used herein can be, for example, a magnetic disk drive, a solid-state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk can be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD ROM). The disk can store an operating system that controls or allocates resources of a computing device. 
     A “memory”, as used herein can include volatile memory and/or non-volatile memory. Non-volatile memory can include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory can include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM). The memory can store an operating system that controls or allocates resources of a computing device. 
     A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system. A module may also include logic, a software-controlled microprocessor, a discreet logic circuit, an analog circuit, a digital circuit, a programmed logic device, a memory device containing executing instructions, logic gates, a combination of gates, and/or other circuit components. Multiple modules may be combined into one module and single modules may be distributed among multiple modules. 
     An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications may be sent and/or received. An operable connection may include a wireless interface, a physical interface, a data interface and/or an electrical interface. 
     A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor may include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that may be received, transmitted and/or detected. Generally, the processor may be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor may include various modules to execute various functions. 
     A “vehicle”, as used herein, refers to any moving vehicle that is capable of carrying one or more human occupants and is powered by any form of energy. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, go-karts, amusement ride cars, rail transport, personal watercraft, and aircraft. In some cases, a motor vehicle includes one or more engines. Further, the term “vehicle” may refer to an electric vehicle (EV) that is capable of carrying one or more human occupants and is powered entirely or partially by one or more electric motors powered by an electric battery. The EV may include battery electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). The term “vehicle” may also refer to an autonomous vehicle and/or self-driving vehicle powered by any form of energy. The autonomous vehicle may or may not carry one or more human occupants. Further, the term “vehicle” may include vehicles that are automated or non-automated with pre-determined paths or free-moving vehicles. 
     A “value” and “level”, as used herein may include, but is not limited to, a numerical or other kind of value or level such as a percentage, a non-numerical value, a discrete state, a discrete value, a continuous value, among others. The term “value of X” or “level of X” as used throughout this detailed description and in the claims refers to any numerical or other kind of value for distinguishing between two or more states of X. For example, in some cases, the value or level of X may be given as a percentage between 0% and 100%. In other cases, the value or level of X could be a value in the range between 1 and 10. In still other cases, the value or level of X may not be a numerical value, but could be associated with a given discrete state, such as “not X”, “slightly x”, “x”, “very x” and “extremely x”. 
     I. System Overview 
     Referring now to the drawings, wherein the showings are for purposes of illustrating one or more exemplary embodiments and not for purposes of limiting the same,  FIG.  1    is a schematic view of an exemplary operating environment  100  for providing weakly-supervised online action segmentation according to an exemplary embodiment of the present disclosure. The operating environment  100  includes a system that provides for the computer-implemented execution of weakly-supervised action segmentation methods use an ordered sequence of action labels for video clips. The ordered sequence of action labels is utilized for artificial intelligence training and forgoes using subjective labeling of start and end times of each action. 
     In an exemplary embodiment, the operating environment  100  may include an externally hosted server infrastructure (external server)  102  that is configured to execute an online action segmentation application  104 . As discussed in more detail below, the online action segmentation application  104  may be configured to perform weakly-supervised online action segmentation with respect to one or more procedures. 
     The one or more procedures may include, but may not be limited to, assembly line procedures, cooking procedures, building procedures, manufacturing procedures, mechanical procedures, vehicular autonomous control procedures, procedural steps, and the like. Each respective procedure may include one or more process steps that may be completed to accomplish an end goal/result of the respective procedure. In an exemplary embodiment, during a training mode of the application  104 , the online action segmentation application  104  may be configured to determine and identify a plurality of atomic actions that may be included as part of each of the process steps that may be completed by one or more individuals (e.g., human), one or more human controlled machines (e.g., human controlled robotic machinery), and/or one or more autonomously controlled apparatuses (e.g., robots, machinery, vehicles, medical tools, automotive tools, etc.). The plurality of atomic actions may be identified as physical, virtual, and/or interactive actions that are sequentially conducted to accomplish an end goal/result of a respective procedure. As discussed below, the plurality of atomic actions may be determined through the analysis of multiple-view (multi-view) video of the respective procedure. 
     Upon determining and identifying the plurality of atomic actions that may be conducted by one or more individuals, the online action segmentation application  104  may be configured to train a neural network  106  during a training mode of the application  104 . The neural network  106  may be trained with data points that are associated with each of a plurality of atomic actions that may conducted (sequentially) to accomplish an end goal/result of a particular procedure. In particular, during the training mode, the online action segmentation application  104  may be configured to utilize multiple image sensors (cameras)  108  that may be located within the training environment  110  to capture multi-view videos of the training environment  110 . 
     The training environment  110  may include, but may not be limited to an environment in which a respective procedure is completed and the end goal/result of the procedure is accomplished and captured by the cameras  108  to train the neural network  106 . The training environment  110  may be an environment that a plurality of atomic actions of a respective procedure are completed to accomplish the end goal/result of the respective procedure to train the neural network  106  using multi-view weakly-supervised training. For example, the training environment  110  may include an assembly line environment, a cook line environment, a manufacturing environment, a workplace environment, a laboratory environment, a simulated environment, and the like in which a respective procedure may take place within the training mode of the online action segmentation application  104  to train the neural network  106  with respect to the plurality of atomic actions. 
     The multi-view videos of the training environment  110  may include a plurality of views of a plurality of atomic actions that may be conducted in order to complete a particular procedure. The online action segmentation application  104  may be configured to utilize multi-view videos to capture procedures with robustness that are less susceptible to issues that pertain to particular camera occlusion, poor lighting conditions, and/or scene variations. The online action segmentation application  104  may also be configured to exploit framewise correspondence between multiple views as supervision for training without requiring pre-labeled datasets. 
     As discussed below, the online action segmentation application  104  may utilize weakly-supervised action segmentation by using an ordered sequence of action labels that pertain to each of the plurality of atomic actions that are determined and identified during the training mode. The online action segmentation application  104  does not require temporal assignment (start and end time) and may avoid the ordering of constraints which are manually intensive, time consuming, and expensive. 
     In one embodiment, the online action segmentation application  104  may be configured to determine and label each of a plurality of atomic actions and data that may associated with each of the plurality of atomic actions that may be included within each of the process steps of the procedure. The determination of each of the plurality of atomic actions may include determining a sequence of the plurality of atomic actions that may be completed to accomplish the procedure. Upon determining each of the plurality of atomic actions, data that may associated with each of the plurality of atomic actions, and the sequence of the plurality of atomic actions that may be completed to accomplish the procedure, the online action segmentation application  104  may be configured to train the neural network  106 . 
     The training of the neural network  106  may be completed by adding data points that are respectively associated with each of the plurality of atomic actions that may be included as part of respective process steps that are required to complete a particular procedure to an action sequencing learning dataset  112  of the neural network  106 . The training of the neural network  106  may also include adding data points that may be associated with the sequence of the plurality of atomic actions that may be completed to accomplish the end goal/result of the procedure to the action sequencing learning dataset  112  of the neural network  106 . 
     The training of the neural network  106  during the training mode of the online action segmentation application  104  may enable the online action segmentation application  104  to implement an execution mode. Within the execution mode, the online action segmentation application  104  may provide artificial intelligence-based guidance, feedback, and/or instructions to allow each respective procedure to be completed with the real-time computer implemented online action understanding of atomic actions that may take place in real time. During the execution mode, the procedure may be executed in real time based on the execution of particular atomic actions by one or more individuals, one or more human controlled machines, and/or one or more autonomously controlled apparatuses towards accomplishment an end goal/result of a particular procedure. Accordingly, the execution mode may involve a real-time implementation of atomic actions that may be conducted by one or more individuals, one or more human controlled machines, and/or one or more autonomously controlled apparatuses to complete process steps that may be involved in the accomplishment of a particular procedure. 
     In one embodiment, the execution mode of the online action segmentation application  104  may be implemented during the real-time implementation of a procedure by one or more individuals, one or more human controlled machines, and/or one or more autonomously controlled apparatuses within an execution environment  114 . The execution environment  114  may include, but may not be limited to an environment in which a respective procedure is completed in real-time and the end goal/result of the procedure is accomplished and captured in real-time by the multiple image sensors (cameras)  116  that may be located within the execution environment  114 . For example, the execution environment  114  may include an assembly line environment, a cook line environment, a manufacturing environment, a workplace environment, a laboratory environment, a simulated environment, and the like in which a respective procedure may take place in real-time subsequent to the training of the neural network  106  by the online action segmentation application  104 . 
     In particular, the online action segmentation application  104  may be configured to receive image data of videos taken of the real-time implementation of the procedure that may be captured by cameras  116  that may be located within the execution environment  114 . The image data may include data associated with single view or multi-view video of the execution environment  114  that captures the implementation of a plurality of real-time atomic actions that may be implemented to complete the particular procedure. 
     In an exemplary embodiment, the online action segmentation application  104  may be configured to execute online action segmentation that may be utilized to determine and identify each particular atomic action based on machine learning deep learning techniques of the trained neural network  106 . In particular, during the execution mode, the online action segmentation application  104  may be configured to determine when and which atomic action(s) in the sequence of the plurality of atomic actions of a respective procedure is started in real-time, is in the process or being completed in real-time, and/or has been completed to provide artificial intelligence-based guidance, feedback, and/or instructions. The functionality of providing artificial intelligence-based guidance, feedback, and/or instructions may enable each respective procedure to be completed with the real-time computer implemented online action understanding of atomic actions towards the accomplishment of an end goal/result of particular procedures as previously trained to the neural network  106  during the training mode of the online action segmentation application  104 . 
     In an exemplary embodiment, the online action segmentation application  104  may be configured to utilize the neural network  106  to utilize machine learning deep learning techniques to use online segmentation to segment video captured on the real-time implementation of one or more atomic actions within the execution environment  114  using dynamic programming without having any access to future frames. In other words, the online segmentation may be completed in real-time without requiring access to and analysis of an entire video that includes past, present, and future frames of an implementation and completion of a particular procedure that may be traditionally be used in an offline setting. Accordingly, the execution of online action segmentation by the online action segmentation application  104  may be utilized to identify particular action labels that may be associated with real-time atomic actions that may take place during the real-time implementation of a particular procedure based on a set of action labels that have been previously trained to the neural network  106  during the training mode using weakly-supervised action segmentation by the online action segmentation application  104 . 
     The online action segmentation application  104  provides an improvement to a computer and technology with respect to video-based action training and identification by enabling a weakly-supervised training of data associated with atomic actions through limited annotation and the determination of real-time data based on the online analysis of action labels. The online action segmentation application  104  may additionally offer dynamic programming to provide weakly-supervised online action segmentation to generate accurate action pseudo-ground-truth in weakly labeled videos without additional annotation cost, utilization of high amount of processing power, and/or high amounts of storage utilization. The functionality of the online action segmentation application  104  allows for minimal training time, identification time, and manual effort on the part of human annotators. The functionality may also eliminate any time, effort, and allocation of resources that may devoted toward offline video recording and offline video playback that may be traditionally be used in an offline setting. 
     In one or more embodiments, upon the identification of each real-time atomic action, the online action segmentation application  104  may be configured to output instructions to electronically control one or more computing systems to provide feedback with respect to real-time atomic actions involved in completing the particular procedure. In one configuration, the online action segmentation application  104  may be configured to control a computing system  118  that may be located within the execution environment  114 . 
     The computing system  118  may be controlled by the online action segmentation application  104  to provide a human machine interface (HMI) feedback to an individual with respect to the completion of a particular atomic action that may be involved in a sequence of a plurality of atomic actions in the completion of a particular procedure. The computing system  118  may also be controlled to provide an HMI that may provide instructions to the individual in real-time regarding a next atomic action that should take place in a particular procedure, one or more anomalies that may be determined as one or more individuals complete respective atomic actions, one or more anomalies that may be determined as one or more human controlled machines complete respective atomic actions, and/or one or more anomalies that may be determined as one or more autonomously controlled apparatuses completes respective atomic actions. Accordingly, the online action segmentation application  104  may provide a user (e.g., individual, entity) with a real-time feedback during the implementation of a procedure to guide, assist, and/or direct the user towards completion of the particular procedure. 
     In another configuration, the online action segmentation application  104  may be configured to control an autonomous apparatus  120  (e.g., robotic apparatus, manufacturing apparatus, vehicular apparatus, medical apparatus, etc.) that may be located within the execution environment  114 . The autonomous apparatus  120  may be controlled to autonomously perform one or more atomic actions that may be required to complete a particular procedure. The autonomous apparatus  120  may be controlled to autonomously perform atomic actions that may start a procedure, continue a procedure based on the trained sequence of atomic actions, and/or complete a procedure based on the trained sequence of atomic actions (e.g., trained during the training mode of the application  104 ). 
     Accordingly, the online action segmentation application  104  may be utilized with respect to the training and providing real-time artificial intelligence-based guidance, feedback, and/or control with respect to the completion of particular atomic actions that are required to complete one or more procedures. As a non-limiting illustrative example, the training mode of the online action segmentation application  104  may be utilized to capture multi-view videos of the training environment  110  that may be configured an assembly line that may be used to complete a vehicle assembly procedure. Each step in the assembly line may include one or more atomic actions that may pertain to the completion the procedure of assembling the vehicle. 
     Within the training mode, the online action segmentation application  104  may be configured to train the neural network  106  with respect to each of the plurality of atomic actions that may pertain to the completion of process steps that are included to complete the vehicle assembly procedure based on the utilization of weakly-supervised action segmentation. The online action segmentation application  104  may use an ordered sequence of action labels that pertain to each of the plurality of atomic actions during the training mode. For non-limiting exemplary purposes, such labels may pertain to sequential order of action labels that pertain to welding of structural pieces to complete a vehicle frame and chassis, attachment of vehicle doors to the vehicle frame, attachment of vehicle body panels, installment of mechanical components, installment of electrical components, and the attachment of wheels and tires to accomplish the end goal of vehicle assembly. 
     During the execution mode, the online action segmentation application  104  may subsequently be utilized to provide one or more commands to the computing system  118  that may be configured as an automation controller to control the autonomous apparatus  120  that may be configured as a mechanical assembly line robotic apparatus. The commands may be based on the real-time identification of atomic actions that may be completed to perform one or more process steps during the execution of the vehicle assembly procedure. The online action segmentation application  104  may send and receive data to and from the neural network  106  to utilize machine learning deep learning techniques to use online segmentation to segment video captured of the real-time implementation of one or more atomic actions of the vehicle assembly procedure. 
     The online action segmentation application  104  may be configured to identify particular action labels that may be associated with real-time atomic actions that may pertain to each of the steps of the vehicle assembly procedure based on a set of action labels that are trained to the neural network  106  (due to the prior addition of data points to the action sequencing learning dataset  112  during the training mode of the online action segmentation application  104 ). Accordingly, upon determining one or more particular atomic actions are completed, the online action segmentation application  104  may be configured to send one or more commands to electronically control the autonomous apparatus  120  to perform one or more subsequent atomic actions to complete subsequent manufacturing steps towards completion of the vehicle assembly procedure. 
     For non-limiting exemplary purposes, upon the completion of an atomic action(s) with respect to the attachment of vehicle doors to the vehicle frame by the autonomous apparatus  120 , the online action segmentation application  104  may utilize online segmentation to segment video captured on the real-time implementation of one or more atomic actions of the vehicle assembly procedure to identify the completion of the atomic action(s) pertaining to the attachment of vehicle doors to the vehicle frame. Accordingly, the online action segmentation application  104  may be configured to send one or more commands to the computing system  118  to control the autonomous apparatus  120  to complete one or more atomic actions such as changing positions, configurations, and/or moving particular components that may be completed to complete the next sequential step of attaching vehicle body panels in real-time. This functionality may allow the autonomous apparatus  120  to be autonomously controlled to perform one or more particular atomic actions required to attach the vehicle body panels to the frame of the vehicle as the next sequential process step towards completion of the end goal/result of vehicle assembly procedure. The online action segmentation application  104  may also be configured to provide a feedback HMI to an individual through the computing system  118  with respect to a real-time status of the plurality of atomic actions based on the trained sequence of atomic actions to complete the vehicle assembly procedure. 
     With continued reference to  FIG.  1   , the external server  102  may be operably controlled by a processor  122  that may be configured to execute the online action segmentation application  104 . In particular, the processor  122  may be configured to execute one or more applications, operating systems, database, and the like. The processor  122  may also include internal processing memory, an interface circuit, and bus lines for transferring data, sending commands, and communicating with the plurality of components of the external server  102 . 
     The processor  122  may be operably connected to a communication unit  124  of the external server  102 . The communication unit  124  may include one or more network interface cards (not shown) that may be configured to connect to one or more computing systems through an internet cloud (not shown). In particular, the communication unit  124  may be configured to provide secure communications between the external server  102 , a computing system(s) (not shown) that may be located within the training environment  110 , the cameras  108  that may be located within the training environment  110 , the computing system  118  that may be located within the execution environment  114 , the cameras  116  that may be located within the execution environment  114 , the autonomous apparatus  120  that may be located within the execution environment  114 , and/or an automation controller (not shown) that may be executed upon the computing system  118  that may be located within the execution environment  114 . 
     The communication unit  124  may be configured to ensure secure communication of data between computing and electronic systems of the training environment  110  to send/receive data to/from the training environment  110  to the online action segmentation application  104  and the neural network  106  stored upon the external server  102  through the internet cloud. Additionally, the communication unit  124  may be configured to ensure secure communication of data between computing and electronic systems of the execution environment  114  to send/receive data to/from the execution environment  114  to the online action segmentation application  104  and the neural network  106  stored upon the external server  102  through the internet cloud. 
     In one embodiment, the processor  122  may be operably connected to a memory  126  of the external server  102 . Generally, the processor  122  may communicate with the memory  126  to execute the one or more applications, operating systems, and the like that are stored within the memory  126 . In one embodiment, the memory  126  may store one or more executable application files that are associated with the online action segmentation application  104 . In an exemplary embodiment, the memory  126  of the external server  102  may be configured to store the neural network  106 . 
     In one embodiment, the neural network  106  may be configured as a convolutional recurrent neural network (CNN). As an CNN, the neural network  106  may execute machine learning/deep learning techniques to process and analyze sequences of data points such as image data associated with multi-view videos that may be captured by the cameras  108  within the training environment  110  and/or the cameras  116  within the execution environment  114 . In an exemplary embodiment, the neural network  106  may be trained based on weakly-supervised action segmentation by using an ordered sequence of action labels that pertain to each of the plurality of atomic actions that are captured in multi-view videos by the cameras  108  within the training environment  110 . 
     The neural network  106  may be trained with data points that are associated with each of a plurality of atomic actions that may conducted (sequentially) to accomplish an end goal/result of a particular procedure. The data points associated with the particular procedure may be trained by populating the datapoints to the action sequencing learning dataset  112  of the neural network  106 . 
     II. The Online Action Segmentation Application and Related Methods 
     Components of the online action segmentation application  104  will now be described according to an exemplary embodiment and with continued reference to  FIG.  1   . In an exemplary embodiment, the online action segmentation application  104  may be stored on the memory  126  and executed by the processor  122  of the external server  102 . In another embodiment, the online action segmentation application  104  may be stored upon a memory of a computing system that may be located within the training environment  110  and/or the computing system  118  that may be located within the execution environment  114  and may be accessed by the communication unit  124  of the external server  102  to be executed by the processor  122  of the external server  102 . 
       FIG.  2    is a schematic overview of the online action segmentation application  104  according to an exemplary embodiment of the present disclosure. In an exemplary embodiment, the online action segmentation application  104  may include a plurality of modules  202 - 208  that may be configured to provide weakly-supervised online action segmentation. The plurality of modules  202 - 208  may include a multi-view image data reception module (data reception module)  202 , a weakly-supervised action segmentation module (action segmentation module)  204 , an online action segmentation module (online segmentation module)  206 , and an application control module  208 . However, it is appreciated that the online action segmentation application  104  may include one or more additional modules and/or sub-modules that are included in lieu of the modules  202 - 208 . 
       FIG.  3    is a process flow diagram of a method  300  for training the neural network  106  with respect to a plurality of atomic actions of a procedure within the training mode of the online action segmentation application  104  according to an exemplary embodiment of the present disclosure.  FIG.  3    will be described with reference to the components of  FIG.  1    though it is to be appreciated that the method  300  of  FIG.  3    may be used with other systems/components. The method  300  may begin at block  302 , wherein the method  300  may include receiving image data from cameras  108  located within the training environment  110 . 
     In one embodiment, the data reception module  202  of the online action segmentation application  104  may be configured to communicate with the cameras  108  to collect image data associated untrimmed videos that are capture multiple-views of the training environment  110  by the cameras  108 . The multi-view videos may be captured at one or more points in time during the starting, duration, and/or ending of a particular procedure (e.g., vehicle assembly procedure, a cooking line procedure, medical procedure, etc.). Accordingly, the multi-view videos may capture one or more atomic actions that may be conducted by one or more individuals, one or more human controlled machines, and/or one or more autonomously controlled apparatuses that are conducted towards accomplishing the end goal/result of the particular procedure. 
     In one embodiment, upon receipt of the image data that are associated with multi-view videos of the procedure being completed within the training environment  110 , the data reception module  202  may be configured to communicate the image data to the action segmentation module  204 . Accordingly, the image data associated with the multi-view videos of the atomic actions being conducted to accomplish the procedure are received by the action segmentation module  204  of the online action segmentation application  104 . 
     The method  300  may proceed to block  304 , wherein the method  300  may include analyzing the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions. In an exemplary embodiment, upon receipt of the image data, the action segmentation module  204  may be configured to analyze the image data using an ordered sequence of action labels that pertain to each of the plurality of atomic actions that are captured within the multi-view videos of the training environment  110 . In particular, the action segmentation module  204  may utilize weakly-supervised online action segmentation of the multi-view videos that do not require the use of labels of start and end times with respect to each of the plurality of atomic actions and of the procedure. 
     In one configuration, within the training mode, the input to the model is multi-view videos of length T represented by frame-level features x 1   T  and an ordered sequence of atomic actions τ=(τ 1 , τ 2  . . . τ M ) known as the transcript. M is the number of atomic actions in a given video and can vary across multi-view videos. Information about the start and end time of each action is not known. Given the set of action labels in the dataset A, the goal of the application  104  is to identify the action label a t  ∈A at frame t for all 0&lt;t&lt;T+1 within the execution mode based on only the past and current observations x 1   t . 
     In one embodiment, in order to fully account for past atomic actions and their duration, the action segmentation module  204  may formulate a marginal causal (or online) probability p on (a t′ |x 1   t ) of the present atomic action a t′ =a n(t′)  at segment n′=n(t′) over all previous actions a 1   n′-1  if n′&gt;1 and duration l 1   n′ . The inferred present atomic action a {circumflex over (t)}  is derived as follows: 
     
       
         
           
             
               α 
               t 
               ′ 
             
             = 
             
               
                 argmax 
                 ⁢ 
                 
                   { 
                   
                     
                       p 
                       on 
                     
                     ( 
                     
                       
                         a 
                         
                           t 
                           ′ 
                         
                       
                       | 
                       
                         x 
                         1 
                         
                           t 
                           ′ 
                         
                       
                     
                     ) 
                   
                   } 
                 
               
               = 
               
                 argmax 
                 ⁢ 
                 
                   { 
                   
                     
                       ∑ 
                       
                         
                           a 
                           1 
                           
                             
                               n 
                               ′ 
                             
                             - 
                             1 
                           
                         
                         , 
                         
                           l 
                           1 
                           
                             n 
                             l 
                           
                         
                       
                     
                     
                       
                         p 
                         on 
                       
                       ( 
                       
                         
                           a 
                           1 
                           
                             n 
                             ′ 
                           
                         
                         , 
                         
                           
                             l 
                             1 
                             
                               n 
                               ′ 
                             
                           
                           | 
                           
                             x 
                             1 
                             
                               t 
                               ′ 
                             
                           
                         
                       
                       ) 
                     
                   
                   } 
                 
               
             
           
         
       
     
     In one embodiment, to improve computational efficiency, the action segmentation module  204  may be configured to empirically approximate the aforementioned equation by the maximum join probability value: 
       a t′ ≈argmax{max p on (a 1   n ,l 1   n |x 1   t′ )}
 
     which may involve two steps. The first is to find the most likely sequence of actions ã 1   n′  with duration {tilde over (l)} 1   n′  until time t′ The second involves taking only the last segment label ã n′ =pop (ã 1   n′ ) to infer the label of the current frame t′, where pop ( ) is a function that outputs the last element of a list. To execute the first step, online inference of the most likely sequence of past action segments (ã 1   n′ , {tilde over (l)} 1   n′ ) is formulated as argmax {p on (a 1   n′ , l 1   n′ |x 1   t′ )}, where p on (a 1   n′ , l 1   n′ |x 1   t     0   ) for n′&gt;1 is derived as: 
     
       
         
           
             
               
                 p 
                 on 
               
               ( 
               
                 
                   
                     a 
                     1 
                     
                       n 
                       ′ 
                     
                   
                   ⁢ 
                   
                     l 
                     1 
                     
                       n 
                       ′ 
                     
                   
                 
                 | 
                 
                   x 
                   1 
                   
                     t 
                     ′ 
                   
                 
               
               ) 
             
             = 
             
               
                 Γ 
                 ⁡ 
                 ( 
                 
                   
                     l 
                     
                       n 
                       ′ 
                     
                   
                   | 
                   
                     a 
                     
                       n 
                       ′ 
                     
                   
                 
                 ) 
               
               ⁢ 
               
                 
                   
                     ∏ 
                     
                       t 
                       = 
                       1 
                     
                   
                   
                     t 
                     ′ 
                   
                 
                 
                   
                     p 
                     ⁡ 
                     ( 
                     
                       
                         x 
                         t 
                       
                       | 
                       
                         a 
                         
                           n 
                           ⁡ 
                           ( 
                           t 
                           ) 
                         
                       
                     
                     ) 
                   
                   ⁢ 
                   
                     
                       
                         ∏ 
                         
                           n 
                           = 
                           1 
                         
                       
                       
                         
                           n 
                           ′ 
                         
                         - 
                         1 
                       
                     
                     
                       
                         p 
                         ⁡ 
                         ( 
                         
                           
                             l 
                             n 
                           
                           | 
                           
                             a 
                             n 
                           
                         
                         ) 
                       
                       · 
                       
                         p 
                         ⁡ 
                         ( 
                         
                           a 
                           1 
                           
                             n 
                             ′ 
                           
                         
                         ) 
                       
                     
                   
                 
               
             
           
         
       
     
     where p(a 1   n′ )=1 if a 1   n′  is a subsequence of any of the transcripts in the training set, and 0 otherwise, and γ(l|a) is a half Poisson function to model the duration l n′  of the current action a n′  at the last observed segment, given by 
     
       
         
           
             
               Γ 
               ⁡ 
               ( 
               
                 l 
                 | 
                 a 
               
               ) 
             
             = 
             { 
           
         
       
       
         
           
             
               
                 1 
                 ⁢ 
                 
                   λ 
                   a 
                   l 
                 
                 ⁢ 
                 
                   exp 
                   ⁡ 
                   ( 
                   
                     - 
                     
                       λ 
                       a 
                     
                   
                   ) 
                 
               
               
                 l 
                 ! 
               
             
             ⁢ 
                 
             if 
             ⁢ 
                 
             1 
             ⁢ 
                 
             otherwise 
           
         
       
     
     where λ a  is the estimated mean length of action a. 
     Inclusion of γ( ) in the online inference of the current action may account for the two following cases. First, using a full Poisson distribution to model the duration of the current observed atomic action leads to penalizing the current atomic actions with a short duration, l n′ &lt;λ a     n′   . Second, γ( ) enables penalization of the current action if its duration is longer than expected since this can be concluded solely based on the observed segment of the atomic action. 
     With continued reference to the method  300  of  FIG.  3   , the method  300  may proceed to block  306 , wherein the method  300  may include training the neural network  106  with data associated with a sequence of the plurality of atomic actions. In an exemplary embodiment, upon analyzing the image data using the weakly-supervised online action segmentation to identify each of the plurality of atomic actions of the respective procedure, the action segmentation module  204  may access the memory  126  of the external server  102  to train the neural network  106 . 
     In one embodiment, the action segmentation module  204  may be configured to access the neural network  106  and may update the action sequencing learning dataset  112  with the data points that are respectively associated with each of the plurality of atomic actions that may be included as part of respective process steps that are required to complete the particular procedure to an action sequencing learning dataset  112  of the neural network  106 . The training of the neural network  106  may also include adding data points that may be associated with the sequence of the plurality of atomic actions that may be completed to accomplish the end goal/result of the procedure to the action sequencing learning dataset  112  of the neural network  106 . 
     A final result that is trained to the neural network  106  as the datapoints that are populated upon the action sequencing learning dataset  112  include a sequence of N predicted segments identified online by their action a n  and duration l n , where n refers to the n th  segment. Accordingly, the neural network  106  may be trained based on weakly-supervised action segmentation by using an ordered sequence of action labels that pertain to each of the plurality of atomic actions that are captured in multi-view videos by the cameras  108  within the training environment  110 . 
       FIG.  4    is a process flow diagram of a method  400  for completing online action segmentation to label atomic actions that are occurring in real-time according to an exemplary embodiment of the present disclosure.  FIG.  4    will be described with reference to the components of  FIG.  1    though it is to be appreciated that the method  400  of  FIG.  4    may be used with other systems/components. The method  400  may begin at block  402 , wherein the method  400  may include receiving image data from the cameras  116  located within the execution environment  114 . 
     In an exemplary embodiment, during the execution mode of the online action segmentation application  104 , the data reception module  202  of the online action segmentation application  104  may be configured to communicate with the cameras  116  within the execution environment  114  to collect image data associated untrimmed videos that are capture multiple-views of the training environment  110  by the cameras  108 . The multi-view videos may be captured at one or more points in time during the starting, duration, and/or ending of a particular procedure (e.g., vehicle assembly procedure, a cooking line procedure, medical procedure, etc.). Accordingly, the multi-view videos may capture one or more atomic actions that may be conducted in real-time within the execution environment  114  by one or more individuals, one or more human controlled machines, and/or one or more autonomously controlled apparatuses that are conducted towards accomplishing the end goal/result of the particular procedure. 
     In one embodiment, upon receipt of the image data that are associated with multi-view videos of the procedure being completed within the execution environment  114 , the data reception module  202  may be configured to communicate the image data to the online segmentation module  206  of the online action segmentation application  104 . Accordingly, the image data associated with the multi-view videos of the real-time atomic actions being conducted to accomplish the procedure within the execution environment  114  are received by the action segmentation module  204  of the online action segmentation application  104 . 
     The method  400  may proceed to block  404 , wherein the method  400  may include completing online action segmentation to label atomic actions that are occurring in real-time. In an exemplary embodiment, during the execution mode, the action segmentation module  204  may be configured to analyze the image data using an ordered sequence of action labels that pertain to each of the plurality of atomic actions that are captured within the multi-view videos of the execution environment  114 . 
     In particular, a final online segmentation result in a streaming video when the current time t′ changes from 1 to any given time T is the sequence of frame-level actions (â1, . . . , âT), where each â t′ ←ã n′ = pop (ã 1   n′ ) is inferred by: 
     
       
         
           
             
               Γ 
               ⁡ 
               ( 
               
                 
                   l 
                   
                     n 
                     ′ 
                   
                 
                 | 
                 
                   a 
                   
                     n 
                     ′ 
                   
                 
               
               ) 
             
             ⁢ 
             
               
                 
                   ∏ 
                   
                     t 
                     = 
                     1 
                   
                 
                 
                   t 
                   ′ 
                 
               
               
                 
                   p 
                   ⁡ 
                   ( 
                   
                     
                       x 
                       t 
                     
                     | 
                     
                       a 
                       
                         n 
                         ⁡ 
                         ( 
                         t 
                         ) 
                       
                     
                   
                   ) 
                 
                 ⁢ 
                 
                   
                     
                       ∏ 
                       
                         n 
                         = 
                         1 
                       
                     
                     
                       
                         n 
                         ′ 
                       
                       - 
                       1 
                     
                   
                   
                     
                       p 
                       ⁡ 
                       ( 
                       
                         
                           l 
                           n 
                         
                         | 
                         
                           a 
                           n 
                         
                       
                       ) 
                     
                     · 
                     
                       p 
                       ⁡ 
                       ( 
                       
                         a 
                         1 
                         
                           n 
                           ′ 
                         
                       
                       ) 
                     
                   
                 
               
             
           
         
       
     
     p=(a 1   n′ )=1 if a 1   n′  is a sub-sequence of any of the transcripts in the training set, and 0 otherwise, and Γ(l|a) is a half Poisson function to model the duration l n′  of the current action a n′  at the last observed segment, given by 
     
       
         
           
             
               Γ 
               ⁡ 
               ( 
               
                 l 
                 | 
                 a 
               
               ) 
             
             = 
             
               { 
               
                 
                   
                     1 
                     
                       
                         
                           λ 
                           a 
                           l 
                         
                         ⁢ 
                         
                           exp 
                           ⁡ 
                           ( 
                           
                             - 
                             
                               λ 
                               a 
                             
                           
                           ) 
                         
                       
                       
                         l 
                         ! 
                       
                     
                   
                   ⁢ 
                       
                   if 
                   ⁢ 
                       
                   l 
                 
                 &lt; 
                 
                   
                     λ 
                     a 
                   
                   ⁢ 
                       
                   otherwise 
                 
               
             
           
         
       
     
     where λ a  is the estimated mean length of action a. 
     The method  400  may proceed to block  406 , wherein the method  400  may include determining completion of each atomic action with respect to the completion of each atomic action in a sequence. In one embodiment, during the execution mode, the action segmentation module  204  may label the current atomic action α n′  that may be occurring towards the achievement of the end goal/result of the procedure in an online fashion without having access to future frames. The action segmentation module  204  may be configured to communicate with the neural network  106  to determine if the current atomic action α n′  is conducted during the starting, duration, and/or ending of a particular procedure (e.g., vehicle assembly procedure, a cooking line procedure, medical procedure, etc.) based on data points that have been previously added to the action sequencing learning dataset  112  that may be associated with the sequence of the plurality of atomic actions that may be completed to accomplish the end goal/result of the respective procedure to the action sequencing learning dataset  112  of the neural network  106 . 
     In an exemplary embodiment, the action segmentation module  204  may be configured to determine if there is a subsequent atomic action in a sequence of the plurality of atomic actions that are required to be conducted towards the accomplishment of an end goal/result of the respective procedure. If it is determined that there is a subsequent atomic action to the current atomic action α n′  that is being conducted in the sequence of the plurality of atomic actions, the action segmentation module  204  may be configured to retrieve data points that are associated with the subsequent atomic action from the neural network  106 . 
     As an illustrative example, during the completion of the vehicle assembly procedure, a sequential order of action labels may include welding of structural pieces to complete a vehicle frame and chassis, attachment of vehicle doors to the vehicle frame, attachment of vehicle body panels, installment of mechanical components, installment of electrical components, and the attachment of wheels and tires to accomplish the end goal of vehicle assembly. If the current atomic action a n′  in the vehicle assembly procedure is installment of electrical components, it maybe determined that the subsequent atomic action in a sequence of the plurality of atomic actions that are required to be conducted towards the accomplishment of an end goal/result of the vehicle assembly procedure includes the attachment of wheels and tires to accomplish the end goal of vehicle assembly. 
     In one or more embodiments, upon retrieving data points that are associated with the atomic action from the neural network  106 , the action segmentation module  204  may be configured to communicate data pertaining to the subsequent atomic action to the application control module  208  of the online action segmentation application  104 . In an exemplary embodiment, the application control module  208  may be configured to communicate one or more commands to the computing system  118  that may be configured as an automation controller to control the autonomous apparatus  120  to autonomously complete the subsequent atomic action towards the accomplishment of an end goal/result of the procedure. 
     Referring again to the aforementioned example, if the subsequent atomic action in a sequence of the plurality of atomic actions that are required to be conducted towards the accomplishment of an end goal/result of the vehicle assembly procedure includes the attachment of wheels and tires to accomplish the end goal of vehicle assembly, the application control module  208  may be configured to send one or more commands to electronically control the autonomous apparatus  120  within the execution environment  114  to perform the subsequent atomic action(s) to the current atomic action of attaching the wheels and tires to complete the vehicle assembly procedure. 
     In another embodiment, in addition to or in lieu of communicating commands to autonomously control the autonomous apparatus  120  to complete a subsequent atomic action(s) towards accomplishing the procedure, the application control module  208  may be configured to send one or more commands to electronically control the computing system  118  to provide feedback to an individual with respect to the completion of the subsequent atomic action(s) that may be involved in a sequence of a plurality of atomic actions in the completion of the procedure. The computing system  118  may also be controlled to provide an HMI that may provide instructions to the individual in real-time regarding a next atomic action that should take place in a particular procedure, one or more anomalies that may be determined as one or more individuals complete respective atomic actions, one or more anomalies that may be determined as one or more human controlled machines complete respective atomic actions, and/or one or more anomalies that may be determined as one or more autonomously controlled apparatuses completes respective atomic actions. Accordingly, the online action segmentation application  104  may provide the user with a real-time feedback during the implementation of a procedure to guide, assist, and direct the user towards completion of the particular procedure. 
       FIG.  5    is a process flow diagram of a method  500  for providing weakly-supervised online action segmentation according to an exemplary embodiment of the present disclosure.  FIG.  5    will be described with reference to the components of  FIG.  1    though it is to be appreciated that the method  500  of  FIG.  5    may be used with other systems/components. The method  500  may begin at block  502 , wherein the method  500  may include receiving image data associated with multi-view videos of a procedure. In one embodiment, the procedure involves a plurality of atomic actions. 
     The method  500  may proceed to block  504 , wherein the method  500  may include analyzing the image data using weakly-supervised action segmentation to identify each of the plurality of atomic actions by using an ordered sequence of action labels. The method  500  may proceed to block  506 , wherein the method  500  may include training a neural network  106  with data pertaining to the plurality of atomic actions based on the weakly-supervised action segmentation. The method  500  may proceed to block  508 , wherein the method  500  may include executing online action segmentation to label atomic actions that are occurring in real-time based on the plurality of atomic actions trained to the neural network  106 . In one embodiment, at least one computing system is controlled to provide automation or feedback with respect to real-time atomic actions involved in completing the procedure based on the online action segmentation. 
     It should be apparent from the foregoing description that various exemplary embodiments of the disclosure may be implemented in hardware. Furthermore, various exemplary embodiments may be implemented as instructions stored on a non-transitory machine-readable storage medium, such as a volatile or non-volatile memory, which may be read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a non-transitory machine-readable storage medium excludes transitory signals but may include both volatile and non-volatile memories, including but not limited to read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, and similar storage media. 
     It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in machine readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown. 
     It will be appreciated that various implementations of the above-disclosed and other features and functions, or alternatives or varieties thereof, may be desirably combined into many other different systems or applications. Also, that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.