Package: pv-evaluation
📊 PatentsView-Evaluation: Benchmark Disambiguation Algorithmspv_evaluationis a Python package built to help advance research onauthor/inventor name disambiguationsystems such asPatentsView. It provides:A large set of benchmark datasets for U.S. patents inventor name disambiguation.Disambiguation summary statistics, evaluation methodology, and performance estimators through theER-Evaluation Python package.See theproject websitefor full documentation. TheExamplespage provides real-world examples of the use ofpv_evaluationsubmodules.Submodulespv_evaluationhas the following submodules:benchmark.data: Access to evaluation datasets and standardized comparison benchmarks. The following benchmark datasets are available:Academic Life Sciences (ALS) inventors benchmark.Israeli inventors benchmark.Engineering and Sciences (ENS) inventors benchmark.Lai's 2011 inventors benchmark.PatentsView's 2021 inventors benchmark.Binette et al.'s 2022 inventors benchmark.benchmark.report: Visualization of key monitoring and performance metrics.templates: Templated performance summary reports.InstallationInstall the released version ofpv_evaluationusingpipinstallpv-evaluationRendering reports requires the installation of quarto fromquarto.org.ExamplesNote: Working with the full patent data requires large amounts of memory (we suggest having 64GB RAM available).See the examples page for complete reproducible examples. The examples below only provide a quick overview ofpv_evaluation's functionality.Metrics and Summary StatisticsGenerate an html report summarizing properties of the current disambiguation algorithm (seethis example):frompv_evaluation.templatesimportrender_inventor_disambiguation_reportrender_inventor_disambiguation_report(".",disambiguation_files=["disambiguation_20211230.tsv","disambiguation_20220630.tsv"],inventor_not_disambiguated_file="g_inventor_not_disambiguated.tsv")Benchmark DatasetsAccess PatentsView-Evaluation's large collection of benchmark datasets:frompv_evaluation.benchmarkimport*load_lai_2011_inventors_benchmark()load_israeli_inventors_benchmark()load_patentsview_inventors_benchmark()load_als_inventors_benchmark()load_ens_inventors_benchmark()load_binette_2022_inventors_benchmark()load_air_umass_assignees_benchmark()load_nber_subset_assignees_benchmark()Representative Performance EvaluationSeethis exampleof how representative performance estimates are obtained from Binette's 2022 benchmark dataset.CitationBinette, Olivier, Sarvo Madhavan, Jack Butler, Beth Anne Card, Emily Melluso and Christina Jones. (2023).PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation.arXiv e-prints: arxiv:2301.03591Binette, Olivier, Sokhna A York, Emma Hickerson, Youngsoo Baek, Sarvo Madhavan, Christina Jones. (2022).Estimating the Performance of Entity Resolution Algorithms: Lessons Learned Through PatentsView.org.arXiv e-prints: arxiv:2210.01230ContributingContribute code and documentationLook through theGitHub issuesfor bugs and feature requests. To contribute to this package:Fork this repositoryMake your changes and update CHANGELOG.mdSubmit a pull requestFor maintainers: if needed, update the "release" branch and create a release.A conda environment is provided for development convenience. To create or update this environment, make sure you have conda installed and then runmake env. You can then activate the development environment usingconda activate pv-evaluation.The makefile provides other development utilities such asmake blackto format Python files,make datato re-generate benchmark datasets from raw data located on AWS S3, andmake docsto generate the documentation website.Raw dataRaw public data is located on PatentsView's AWS S3 server athttps://s3.amazonaws.com/data.patentsview.org/PatentsView-Evaluation/data-raw.zip. This zip file should be updated as needed to reflect datasets provided by this package and to ensure that original data sources are preserved without modification.TestingThe minimal testing requirement for this package is a check that all code executes without error. We recommend placing execution checks in a runnable notebook and using thetestbookpackage for execution within unit tests. User examples should also be provided to exemplify usage on real data.Report bugs and submit feedbackReport bugs and submit feedback athttps://github.com/PatentsView/PatentsView-Evaluation/issues.