Patent ID: 11915123
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 12:
13. A method comprising:
receiving, by a computing device, a multi-modal data set, the multi-modal data set comprising data in different formats from two or more modalities, including a first data set having a first modality and a second data set having a second modality that differs from the first modality;
separately processing the first and second data sets, including encoding the first data set into first encodings comprising one or more first vectors and encoding the second data set into encodings comprising one or more second vectors;
analyzing the processed multi-modal data set, including exploring inter-relation of information from at least two temporal sequences of a same or different sampling frequency, enabling direct incorporation of semantic information learned from the first modality of the first data feed to every time step reasoning on behaviors of the second modality of the second data feed, iteratively and asynchronously fusing the first and second encodings, the fusing comprising combining first and second vectors representing correlated temporal behavior; and
returning the fused vectors as output data.