Patent ID: 11948348
Assignee: 5DT, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G  H

Claim 18:
19. A The machine-implemented method for monitoring operator behavior, the method comprising:
receiving onto an operator behavior recognition system via an input/output interface onto a processor at least one image from an image source, the operator behavior recognition system comprising hardware including at least one processor, a data storage facility in communication with the processor and one or more input/output interfaces in communication with the processor, the system being configured to implement a set of convolutional neural networks (CNNs) including:
an object detection group into which at least one image is received from an image source for detecting at least one object in the image and to delineate the object from the image for further processing;
a feature extraction group which extracts features of the at least one object detected by the object detection group, the features including key points, orientation, descriptors and other features of the at least one object;
a classifier group which assess the features extracted by the feature extraction group and classifying the features into a predefined number of events and which is operable to report the events to a remote computer, wherein the classifier group includes an ensemble function to ensemble the outputs of the classifiers together with the output of the single image CNN of the at least one object together with the combination of the single image CNN and the LSTM recurrent network by a weighted sum of the classifiers where the weights are determined by optimizing the weights on the training dataset;

processing the at least one image by the object detection group to detect at least one object in the image and to delineate the object from the image for further processing;
processing the at least one image of at least one object via the features extraction group to extract features of the at least one object;
processing an output from the feature extraction group via the classifier group to detect a number of predefined events and transmitting the predefined events to a server;
receiving, on the server, the predefined events detected by the operator behavior recognition system;
displaying one or more events detected by the operator behavior recognition system (detected events) on a web frontend; and
recording, on a database in communication with the server, data and the detected events, wherein the database records all data being generated by the operator behavior recognition system.