Patent ID: 11971963
Assignee: QUANTUM-SI INCORPORATED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method for predicting an association between input data in a first modality and data in a second modality, the method comprising:
accessing a statistical model trained to represent links between data having a plurality of modalities including the first modality and the second modality, the statistical model comprising a plurality of encoders and decoders, each of which is trained to process data for one of the plurality of modalities, and a joint-modality representation coupling the plurality of encoders and decoders, the plurality of encoders and decoders including an encoder/decoder pair for each of the plurality of modalities, wherein:
the encoder of the encoder/decoder pair is trained to project data for the modality into a latent representation space associated with the join-modality representation; and
the decoder of the encoder/decoder pair is trained to decode input from the common representation space to generate output in a representation space of the modality;

selecting, based on the first modality and the second modality, an encoder/decoder pair or a pair of encoders, from among the plurality of encoders and decoders; and
predicting the association between the input data in the first modality and the data in the second modality, the predicting comprising:
processing the input data with the joint-modality representation and the selected encoder/decoder pair or pair of encoders to obtain a projection of the input data in a representation space of the model, the representation space comprising a representation space of the second modality or the latent representation space associated with the joint-modality representation; and
predicting the association between the input data in the first modality and the data in the second modality by determining, in the representation space, a measure of similarity between the projection of the input data and the data in the second modality.