Patent Document ID: 9830516
Application ID: 15203862
Patent Flag: 1

Claim One:
1. A computer-implemented method for joint temporal segmentation and classification of user activities in an egocentric video, the computer-implemented method comprising: receiving, using a data input module on a computer with a processor and a memory, a live dataset including an egocentric video including at least one egocentric video sequence having a plurality of egocentric video frames; extracting, using a feature extraction module on the computer in communication with a joint segmentation and classification (JSC) module on the computer, a plurality of low-level features from the live dataset based on predefined feature categories; determining, using the JSC module on the computer, at least one activity change frame from the plurality of egocentric video frames based on the extracted plurality of low-level features; dividing, using the JSC module on the computer, the live dataset into a plurality of partitions based on the determined at least one activity change frame, wherein each of the plurality of partitions begins with a candidate video frame; computing, using the JSC module on the computer, a recursive cost function at the candidate video frame of each of the plurality of partitions based on dynamic programming; determining, using the JSC module on the computer, a beginning time instant of the candidate frame based on the computation; segmenting, using the JSC module on the computer, the live dataset into a plurality of segments based on the determined beginning time instant; identifying, using the JSC module on the computer, at least one activity segment that corresponds to at least one user activity among the plurality of segments using a predefined activity model being trained based on a multiple instance learning (MIL) based classifier; simultaneously associating, using the JSC module on the computer, a predefined activity label with the identified at least one activity segment; and outputting, using the computer, the live dataset assigned with the predefined activity label for the at least one activity segment.