Package: matchain
MatChain: Simple, Flexible, EfficientMatChain is an experimental package designed for record linkage. Record linkage is the process of matching records that correspond to the same real-world entity in two or more datasets. This process typically includes several steps, such as blocking and the final matching decision, with a wide range of methods available, including probabilistic, rule-based, and machine learning approaches.MatChain was created with three core objectives in mind: simplicity, flexibility, and efficiency. It focuses on unsupervised approaches to minimize manual efforts, allows for customization of matching steps, and offers fast and resource-efficient implementations.MatChain leverages libraries like Pandas, NumPy, and SciPy forvectorized data handling,advanced indexing, and support forsparse matrices.
It also utilizes scikit-learn and SentenceTransformers to convert strings into sparse vectors and dense vectors, respectively. This allows to perform blocking as approximate nearest neighbour search in the resulting set of vectors utilizing libraries likeNMSLIBandFaiss.The currently published version of MatChain exclusively providesAutoCalas the matching algorithm. AutoCal is an unsupervised method initially designed for instance matching withOntomatchin the context ofThe World Avatar.
MatChain's implementation is highly efficient and allows
for the combination of AutoCal with various procedures for blocking and computing similarity scores.InstallationMatChain requires Python 3.8 or higher and can be installed with pip:pip install matchainHowever, this only installs PyTorch's CPU version. If you want to use the GPU version, you need to install it separately:pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118Basic Example Using the APIIn this example, we demonstrate how to match two datasets, denoted as A and B, based on columns with the same names: "year," "title," "authors," and "venue." You can run this example in the accompanying notebookrun_matchain_api.ipynb, which provides a detailed explanation of MatChain's API, including how to specify parameters.First, we read the data and initialize an instance of the classMatChainusing Pandas' dataframes.data_dir='./data/Structured/DBLP-ACM'dfa=pd.read_csv(f'{data_dir}/tableA.csv')dfb=pd.read_csv(f'{data_dir}/tableB.csv')mat=matchain.api.MatChain(dfa,dfb)Next, we specify one or more similarity functions for each matching column by thepropertymethod. These similarity functions calculate scores between 0 and 1 for pairs of column values. In this example, we useequalfor the integer-valued "year" column, which returns 1 if two years are equal and 0 otherwise. For each of the remaining string-valued columns, we applyshingle_tfidfto generate a sparse vector for each string based on its shingles (n-grams on the character level) and compute the cosine similarity between the sparse vectors for pairs of strings:mat.property('year',simfct='equal')mat.property('title',simfct='shingle_tfidf')mat.property('authors',simfct='shingle_tfidf')mat.property('venue',simfct='shingle_tfidf')As the total number of record pairs grows with the product of the record sizes in datasets A and B, classifying each pair as matching or non-matching can be computationally expensive, especially for large datasets. Blocking effectively reduces the number of pairs while only discarding a small fraction of true matching pairs. The following line specifies three columns to use for blocking. By default, MatChain utilizes the librarysparsedottopnto perform blocking by conducting a nearest neighbor search on the same shingle vectors mentioned earlier:mat.blocking(blocking_props=['title','authors','venue'])Finally, we callautocalto execute the matching algorithm AutoCal andpredictto get the predicted matching pairs:mat.autocal()predicted_matches=mat.predict()Configuration FileWhile the example above demonstrates how to use MatChain's API to match two datasets, an alternative and streamlined approach is to utilize a configuration file. This method allows us to specify datasets, matching chains, and parameters in a separate file:python matchain --config ./config/mccommands.yamlFor more detailed information about configuration options, run the notebookrun_matchain_config.ipynb.DatasetsThe data subdirectory includes pairs of example datasets and ground truth data for evaluating MatChain's performance. These datasets cover various domains, including restaurant, bibliography, product, and powerplants. Specifically, four of them originate fromthis paperand were downloaded from theDeepMatcher Data Repository. Two additional dataset pairs are related to the powerplants domain and were originally used forAutoCal.