Patent ID: 11915123
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 6:
7. A computer program product to employ deep learning techniques to fuse data across modalities, the computer program product comprising:
a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
receive a multi-modal data set, the multi-modal data set comprising data in different formats from two or more modalities, including a first data set having a first modality and a second data set having a second modality that differs from the first modality;
separately process the first and second data sets, including encode the first data set into first encodings comprising one or more first vectors and encode the second data set into second encodings comprising one or more second vectors;
analyze the processed multi-modal data set, including exploring inter-relation of information from at least two temporal sequences of a same or different sampling frequency, enabling direct incorporation of semantic information learned from the first modality of the first data feed to every time step reasoning on behaviors of the second modality of the second data feed, iteratively and asynchronously fuse the first and second encodings, the fusion comprising combining the first and second vectors representing correlated temporal behavior; and
return the fused vectors as output data.