Patent ID: 8923630

Claim:
A data mining method, comprising: receiving a set of multimodal data objects comprising semantically interrelated information of a first type and a second type, each being of a different type selected from the group consisting of image information, audio information, video information, and semantic information; representing at least the first type of information of the multimodal data objects as feature vectors within a feature space comprising the first type of information and the second type of information, and the semantic interrelation between the first type of information and the second type of information; clustering the feature vectors into classified clusters according to at least one semantic clustering criterion by at least one automated processor, to thereby determine a classification of the respective feature vectors; associating data objects with respective members of the set of multimodal data objects by the at least one automated processor, based on the clustering, the associated data objects comprising information of a third type semantically interrelated to the second type of information, selected from the group consisting of images, audio, video and semantic information, wherein the type of information of the third type is distinct from the type of information of the first type; estimating a joint feature representation of the set of multimodal data objects and the associated data objects by the at least one automated processor; optimizing the joint feature representation by the at least one automated processor to provide a structured output space of interdependent objects, based on at least a prediction error criterion, by iteratively solving a dual problem by selectively partitioning data objects into a working set and a non-working set, comprising: moving the data objects in the non-working set that can be moved without changing an objective function to the working set, and moving the data objects in the working set that can be moved with a decrease in the objective function to the non-working set; receiving a query represented according to the first type of information; and identifying data objects from the set of multimodal data objects that correspond to the query by the at least one automated processor, based on at least the structured output space of interdependent multimodal objects.