Package: mmkit-features
MMKit-Features: Multimodal Feature Extraction ToolkitTraditional knowledge graphs (KGs) are usually comprised of entities, relationships, and attributes. However, they are not designed to effectively store or represent multimodal data. This limitation prevents them from capturing and integrating information from different modes of data, such as text, images, and audio, in a meaningful and holistic way.TheMMKit-Featuresproject proposes a multimodal architecture to build multimodal knowledge graphs with flexible multimodal feature extraction and dynamic multimodal concept generation.Project GoalTo extract, store, and fuse various multimodal features from multimodal datasets efficiently;To achieve generative adversarial network(GAN)-based multimodal knowledge representation dynamically in multimodal knowledge graphs;To provide a common deep learning-based architecture to enhance multimodal knowledge reasoning in real life.InstallationYou can install this toolkit using ourPyPipackage.pip install mmkit-featuresDesign Science FrameworkFigure 1: Multimodal Computational SequenceFigure 2: GAN-based Multimodal Concept GenerationModalitiesText/Language modalityImage modalityVideo modalityAudio modalityCross-modality among aboveUsageA toy example showing how to build a multimodal feature (MMF) library is here:frommmkfeatures.fusion.mm_features_libimportMMFeaturesLibfrommmkfeatures.fusion.mm_features_nodeimportMMFeaturesNodeimportnumpyasnpif__name__=="__main__":# 1. create an empty multimodal features library with root and dataset namesfeature_lib=MMFeaturesLib(root_name="test features",dataset_name="test_features")# 2. set short names for each dimension for conveniencefeature_lib.set_features_name(["feature1","feature2","feature3"])# 3. set a  list of content IDscontent_ids=["content1","content2","content3"]# 4. according to IDs, assign a group of features with interval to corresponding content IDfeatures_dict={}foridincontent_ids:mmf_node=MMFeaturesNode(id)mmf_node.set_item("name",str(id))mmf_node.set_item("features",np.array([[1,2,3]]))mmf_node.set_item("intervals",np.array([[0,1]]))features_dict[id]=mmf_node# 5. set the library's datafeature_lib.set_data(features_dict)# 6. save the features to disk for future usefeature_lib.save_data("test6_feature.csd")# 7. check structure of lib file with the format of h5pyfeature_lib.show_structure("test6_feature.csd")# 8. have a glance of features content within the datasetfeature_lib.show_sample_data("test6_feature.csd")# 9. Finally, we construct a simple multimodal knowledge base.Further instructions on the toolkit refers tohere.ApplicationsHere are some examples of using our work in real life with codes and documents.1. Multimodal Features ExtractorsText Features ExtractionSpeech Features ExtractionImage Features ExtractoinVideo Features ExtractionTransformer-based Features Extraction2. Multimodal Feature Library (MMFLib)Basic Computational SequenceCore use of MMFLib3. Multimodal Knowledge BasesMultimodal Birds Feature LibraryMultimodal Disease Coding Feature LibraryMultimodal ROCO Feature Library4. Multimodal Indexing and QueryingBrute Force IndexingInverted IndexingPositional IndexingMultimodal Indexing and queryingCreditsThe project includes some source codes from various open-source contributors. Here is a list of their contributions.A2Zadeh/CMU-MultimodalSDKaishoot/Speech_Feature_Extractionantoine77340/video_feature_extractorjgoodman8/py-image-features-extractorv-iashin/Video FeaturesLicenseThemmkit-featuresproject is provided byDonghua Chenwith MIT license.CitationPlease cite our project if the project is used in your research.Chen, D. (2023). MMKit-Features: Multimodal Features Extraction Toolkit (Version 0.0.2) [Computer software]