Patent Document ID: 9830516
Application ID: 15203862

Base Claim:
1. A computer-implemented method for joint temporal segmentation and classification of user activities in an egocentric video, the computer-implemented method comprising: receiving, using a data input module on a computer with a processor and a memory, a live dataset including an egocentric video including at least one egocentric video sequence having a plurality of egocentric video frames; extracting, using a feature extraction module on the computer in communication with a joint segmentation and classification (JSC) module on the computer, a plurality of low-level features from the live dataset based on predefined feature categories; determining, using the JSC module on the computer, at least one activity change frame from the plurality of egocentric video frames based on the extracted plurality of low-level features; dividing, using the JSC module on the computer, the live dataset into a plurality of partitions based on the determined at least one activity change frame, wherein each of the plurality of partitions begins with a candidate video frame; computing, using the JSC module on the computer, a recursive cost function at the candidate video frame of each of the plurality of partitions based on dynamic programming; determining, using the JSC module on the computer, a beginning time instant of the candidate frame based on the computation; segmenting, using the JSC module on the computer, the live dataset into a plurality of segments based on the determined beginning time instant; identifying, using the JSC module on the computer, at least one activity segment that corresponds to at least one user activity among the plurality of segments using a predefined activity model being trained based on a multiple instance learning (MIL) based classifier; simultaneously associating, using the JSC module on the computer, a predefined activity label with the identified at least one activity segment; and outputting, using the computer, the live dataset assigned with the predefined activity label for the at least one activity segment.

---

Claim 3:
3. The computer-implemented method according to claim 1 , further comprising: receiving, using a training module on the computer via the data input module, a training dataset including an egocentric video, wherein the training dataset is pre-segmented based on the predefined activity label that corresponds to the at least one user activity having a predefined time length; extracting, using the feature extraction module, a plurality of low-level features from the training dataset based on the predefined feature categories; partitioning, using the training module, the training dataset into a plurality of overlapping segments of predefined durations; defining, using the training module, a plurality of subsets, each including a distinct combination of overlapping segments from the plurality of overlapping segments based on a duration of the distinct combination being less than half of the predefined time length; determining, using the training module, a set of low-level features that belong to the plurality of subsets, wherein the set is derived from the extracted plurality of low-level features; grouping, using the training module, the plurality of subsets into one or more clusters based on the determined set of low-level features, wherein the one or more clusters include at least one centroid that forms a bag including a plurality of instances; comparing, using the training module, each of the plurality of instances with the predefined activity label; assigning, using the training module, the bag with a positive value if the comparison is positive for at least one of the plurality of instances, or with a negative value if the comparison is negative for all the one or more instances; and training, using the training module, the predefined activity model using the assigned value of the bag based on the multiple instance learning (MIL) based classifier, wherein the activity model corresponds to the at least one user activity in the training dataset.