Package: opendataval
OpenDataVal: a Unified Benchmark for Data ValuationAssessing the quality of individual data points is critical for improving model performance and mitigating biases. However, there is no way to systematically benchmark different algorithms.OpenDataValis an open-source initiative that with a diverse array of datasets/models (image, NLP, and tabular), data valuation algorithms, and evaluation tasks using just a few lines of code.OpenDataValalso provides a leaderboards for data evaluation tasks. We've curated and added
artificial noise to some datasets. Create your ownDataEvaluatorto top theleaderboards.OpenDataValis accepted at NeurIPS 2023 track on Datasets and Benchmarks.OverviewPaperPaper linkPythonDependenciesDocumentationCI/CDIssuesLicenseReleasesCitationCite Us:sparkles: FeaturesFeatureStatusLinksNotesDatasetsStableDocsEmbeddings available for image/NLP datasetsModelsStableDocsSupport available for sk-learn modelsData EvaluatorsStableDocsExperimentsStableDocsExamplesStableCLIExperimentalopendataval --helpNo support for null values(Back to top):hourglass_flowing_sand: Installation optionsInstall with pippipinstallopendatavalClone the repo and installgitclonehttps://github.com/opendataval/opendataval.git
makeinstalla. Install optional dependencies if you'recontributingmakeinstall-devb. If you want to pull in kaggle datasets, I'd reccomend looking how to add a kaggle folder to the current directory. Tutorialhere(Back to top):zap: Quick StartTo set up an experiment on DataEvaluators. Feel free to change the source code as needed for a project.importopendatavalfromopendataval.experimentimportExperimentMediatorfromopendataval.datavalimportDataOobfromopendataval.experimentimportdiscover_corrupted_sample,noisy_detectionexper_med=ExperimentMediator.model_factory_setup(dataset_name='iris',force_download=False,train_count=50,valid_count=50,test_count=50,model_name='ClassifierMLP',train_kwargs={'epochs':5,'batch_size':20},)list_of_data_evaluators=[DataOob()]# Define evaluators hereeval_med=exper_med.compute_data_values(list_of_data_evaluators)# Runs a discover the noisy data experiment for each DataEvaluator and plotsdata,fig=eval_med.plot(discover_corrupted_sample)# Runs non-plottable experimentdata=eval_med.evaluate(noisy_detection):computer: CLIopendatavalcomes with a quickCLItool, The tool is under development and the template for a csv input is found atcli.csv. Note that for kwarg arguments, the input must be valid json.To use run the following command if installed with make-install:opendataval--filecli.csv-n[job_id]-o[path/to/output/]To run without installing the script:python opendataval --file cli.csv -n [job_id] -o [path/to/output/](Back to top):control_knobs: APIHere are the 4 interacting parts of opendatavalDataFetcher, Loads data and holds meta data regarding splitsModel, trainable prediction model.DataEvaluator, Measures the data values of input data point for a specified model.ExperimentMediator, facilitates experiments regarding data values across severalDataEvaluators(Back to top)DataFetcherThe DataFetcher takes the name of aRegisterdataset and loads, transforms, splits, and adds noise to the data set.fromopendataval.dataloaderimportDataFetcherDataFetcher.datasets_available()# ['dataset_name1', 'dataset_name2']fetcher=DataFetcher(dataset_name='dataset_name1')fetcher=fetcher.split_dataset_by_count(70,20,10)fetcher=fetcher.noisify(mix_labels,noise_rate=.1)x_train,y_train,x_valid,y_valid,x_test,y_test=fetcher.datapoints(Back to top)ModelModelis the predictive model for Data Evaluators.fromopendataval.modelimportLogisticRegressionmodel=LogisticRegression(input_dim,output_dim)model.fit(x,y)model.predict(x)>>>torch.Tensor(...)(Back to top)DataEvaluatorWe have a catalog ofDataEvaluatorto run experiments. To do so, input theModel,DataFetcher, and an evaluation metric (such as accuracy).fromopendataval.dataval.ameimportAMEdataval=(AME(num_models=8000).train(fetcher=fetcher,pred_model=model,metric=metric))data_values=dataval.data_values# Cached valuesdata_values=dataval.evaluate_data_values()# Recomputed values>>>np.ndarray([.888,.132,...])(Back to top)ExperimentMediatorExperimentMediatoris helps make a cohesive and controlled experiment. NOTE Warnings are raised if errors occur in a specificDataEvaluator.expermed=ExperimentrMediator(fetcher,model,train_kwargs,metric_name).compute_data_values(data_evaluators)Run experiments by passing in an experiment function:(DataEvaluator, DataFetcher, ...) - > dict[str, Any]. There are 5 foundexper_methods.pywith three being plotable.df=expermed.evaluate(noisy_detection)df,figure=expermed.plot(discover_corrupted_sample)For more examples, please refer to theDocumentation(Back to top):medal_sports: opendataval LeaderboardsFor datasets that start with the prefix challenge, we provideleaderboards. Compute the data values with anExperimentMediatorand use thesave_datavalfunction to save a csv. Upload it tohere! Uploading will allow us to systematically compare yourDataEvaluatoragainst others in the field.Theavailable challengesare currently:challenge-irisexper_med=ExperimentMediator.model_factory_setup(dataset_name='challenge-...',model_name=model_name,train_kwargs={...},metric_name=metric_name)exper_med.compute_data_values([custom_data_evaluator]).evaluate(save_dataval,save_output=True)(Back to top):wave: ContributingIf you have a quick suggestion, reccomendation, bug-fixes please open anissue.
If you want to contribute to the project, either through data sets, experiments, presets, or fix stuff, please see ourContribution page.Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(Back to top):bulb: Visionclean, descriptive specification syntax-- based on modern object-oriented design principles for data science.fair model assessment and benchmarking-- Easily build and evaluate your Data Evaluatorseasily extensible-- Easily add your own data sets,(Back to top):classical_building: LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(Back to top)Cite UsIf you found the library or the paper useful, please cite us!@article{
    jiang2023opendataval,
    title={OpenDataVal: a Unified Benchmark for Data Valuation},
    author={Kevin Fu Jiang and Weixin Liang and James Zou and Yongchan Kwon},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    year={2023},
    url={https://openreview.net/forum?id=eEK99egXeB}
}(Back to top)