Patent ID: 11915123
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer system comprising:
a processor operatively coupled to memory; and
an artificial intelligence (AI) platform, in communication with the processor, having machine learning (ML) tools to employ deep learning techniques operable to fuse data across modalities, the tools comprising:
a first data feed manager operatively coupled to a first data set, the first data set having a first modality in a first data format;
a second data feed manager operatively coupled to a second data set, the second data set having a second modality in a second data form, the second modality being different from the first modality;
the first data feed manager configured to encode the first data set into first encodings comprising a first set of vectors;
the second data feed manager configured to encode the second data set into second encodings comprising a second set of vectors; and
an analyzer operatively coupled to the first and second data feed managers, the analyzer configured to:
explore inter-relation of information from at least two temporal sequences of a same or different sampling frequency;
enable direct incorporation of semantic information learned from the first modality of the first data feed to every time step reasoning on behaviors of the second modality of the second data feed;
leverage an artificial recurrent neural network (RNN) to analyze the encoded first and second data sets, including iteratively and asynchronously fuse the first and second encodings, the fusing comprising combining vectors from the first and second sets of vectors representing correlated temporal behavior; and
return the fused vectors as output data.