Package: ws-benchmark
🔧 New2/13/22Addscriptto generate LFs for any tabular dataset as well as 5 new tabular datasets, namely, mushroom, spambase, PhishingWebsites, Bioresponse, and bank-marketing.11/04/21(beta) Addparallel_fitfor torch model to support pytorch DistributedDataParallel-example10/15/21A branch of new methods: WeaSEL, ImplyLoss, ASTRA, MeanTeacher, Meta-Weight-Net, Learning-to-ReweightSupport image classification (dataset class / torchvision backbone) as well as DomainNet/Animals-with-Attributes2 datasets (check out thedatasetsfolder)🔧 What is it?Wrenchis abenchmark platformcontaining diverse weak supervision tasks. It also provides acommon and easy frameworkfor development and evaluation of your own weak supervision models within the benchmark.For more information, checkout our publications:WRENCH: A Comprehensive Benchmark for Weak Supervision(NeurIPS 2021)If you find this repository helpful, feel free to cite our publication:@inproceedings{
zhang2021wrench,
title={{WRENCH}: A Comprehensive Benchmark for Weak Supervision},
author={Jieyu Zhang and Yue Yu and Yinghao Li and Yujing Wang and Yaming Yang and Mao Yang and Alexander Ratner},
booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2021},
url={https://openreview.net/forum?id=Q9SKS5k8io}
}🔧 What is weak supervision?Weak Supervisionis a paradigm for automated training data creation without manual annotations.For a brief overview, please check out thisblog.For more context, please check out thissurvey.To track recent advances in weak supervision, please follow thisrepo.🔧 Installation[1] Install anaconda:
Instructions here:https://www.anaconda.com/download/[2] Clone the repository:git clone https://github.com/JieyuZ2/wrench.git
cd wrench[3] Create virtual environment:conda env create -f environment.yml
source activate wrenchIf this not working or you want to use only a subset of modules of Wrench, check out thiswiki page🔧 Available DatasetsThe datasets can be downloaded viathis.Note that some datasets may have more training examples than what is reported in README/paper because we include the dev set, whose indices can be found in labeled_id.json if exists.A documentation of dataset format and usage can be found in thiswiki-pageclassification:NameTask# class# LF# train# validation# testdata sourceLF sourceCensusincome clasification28310083556116281linklinkYoutubespam clasification2101586120250linklinkSMSspam clasification2734571500500linklinkIMDBsentiment clasification282000025002500linklinkYelpsentiment clasification283040038003800linklinkAGNewstopic clasification49960001200012000linklinkTRECquestion classification6684965500500linklinkSpouserelation classification292225428012701linklinkSemEvalrelation classification91641749200692linklinkCDRbio relation classification23384309204673linklinkChemprotchemical relation classification10261286116071607linklinkCommercialvideo frame classification246413094797496linklinkTennis Rallyvideo frame classification2669597461098linklinkBasketballvideo frame classification241797010641222linklinkDomainNetimage classification-----linklinksequence tagging:Name# class# LF# train# validation# testdata sourceLF sourceCoNLL-034161404132503453linklinkWikiGold4161355169170linklinkOntoNotes 5.01817115812500022897linklinkBC5CDR29500500500linklinkNCBI-Disease155929999linklinkLaptop-Review132436609800linklinkMIT-Restaurant81671595001521linklinkMIT-Movies12792415002441linklinkThe detailed documentation is coming soon.🔧 Available ModelsIf you find any of the implementations is wrong/problematic, don't hesitate to raise issue/pull request, we really appreciate it!TODO-list: checkthisout!classification:ModelModel TypeReferenceLink to WrenchMajority VotingLabel Model--linkWeighted Majority VotingLabel Model--linkDawid-SkeneLabel ModellinklinkData ProgammingLabel ModellinklinkMeTaLLabel ModellinklinkFlyingSquidLabel ModellinklinkLogistic RegressionEnd Model--linkMLPEnd Model--linkBERTEnd ModellinklinkCOSINEEnd ModellinklinkDenoiseJoint ModellinklinkWeaSELJoint Modellinklinksequence tagging:ModelModel TypeReferenceLink to WrenchHidden Markov ModelLabel ModellinklinkConditional Hidden Markov ModelLabel ModellinklinkLSTM-CNNs-CRFEnd ModellinklinkBERT-CRFEnd ModellinklinkLSTM-ConNetJoint ModellinklinkBERT-ConNetJoint Modellinklinkclassification-to-sequence-tagging wrapper:Wrench also provides aSeqLabelModelWrapperthat adaptes label model for classification task to sequence tagging task.methods from related domains:Robust Learning methods as end model:ModelModel TypeReferenceLink to WrenchMeta-Weight-NetEnd ModellinklinkLearning2ReWeightEnd ModellinklinkSemi-Supervised Learning methods as end model:ModelModel TypeReferenceLink to WrenchMeanTeacherEnd ModellinklinkWeak Supervision with cleaned labels (Semi-Weak Supervision):ModelModel TypeReferenceLink to WrenchImplyLossJoint ModellinklinkASTRAJoint Modellinklink🔧  Quick examples🔧  Label model with parallel grid search for hyper-parametersimportloggingimportnumpyasnpimportpprintfromwrench.datasetimportload_datasetfromwrench.loggingimportLoggingHandlerfromwrench.searchimportgrid_searchfromwrenchimportlabelmodelfromwrench.evaluationimportAverageMeter#### Just some code to print debug information to stdoutlogging.basicConfig(format='%(asctime)s-%(message)s',datefmt='%Y-%m-%d%H:%M:%S',level=logging.INFO,handlers=[LoggingHandler()])logger=logging.getLogger(__name__)#### Load datasetdataset_home='../datasets'data='youtube'train_data,valid_data,test_data=load_dataset(dataset_home,data,extract_feature=False)#### Specify the hyper-parameter search space for grid searchsearch_space={'Snorkel':{'lr':np.logspace(-5,-1,num=5,base=10),'l2':np.logspace(-5,-1,num=5,base=10),'n_epochs':[5,10,50,100,200],}}#### Initialize label modellabel_model_name='Snorkel'label_model=getattr(labelmodel,label_model_name)#### Search best hyper-parameters using validation set in paralleln_trials=100n_repeats=5target='acc'searched_paras=grid_search(label_model(),dataset_train=train_data,dataset_valid=valid_data,metric=target,direction='auto',search_space=search_space[label_model_name],n_repeats=n_repeats,n_trials=n_trials,parallel=True)#### Evaluate the label model with searched hyper-parameters and average metermeter=AverageMeter(names=[target])foriinrange(n_repeats):model=label_model(**searched_paras)history=model.fit(dataset_train=train_data,dataset_valid=valid_data)metric_value=model.test(test_data,target)meter.update(target=metric_value)metrics=meter.get_results()pprint.pprint(metrics)For detailed guidance ofgrid_search, please check outthis wiki page.🔧  Run a standard supervised learning pipelineimportloggingimporttorchfromwrench.datasetimportload_datasetfromwrench.loggingimportLoggingHandlerfromwrench.endmodelimportMLPModel#### Just some code to print debug information to stdoutlogging.basicConfig(format='%(asctime)s-%(message)s',datefmt='%Y-%m-%d%H:%M:%S',level=logging.INFO,handlers=[LoggingHandler()])logger=logging.getLogger(__name__)#### Load datasetdataset_home='../datasets'data='youtube'#### Extract data features using pre-trained BERT model and cache itextract_fn='bert'model_name='bert-base-cased'train_data,valid_data,test_data=load_dataset(dataset_home,data,extract_feature=True,extract_fn=extract_fn,cache_name=extract_fn,model_name=model_name)#### Train a MLP classifierdevice=torch.device('cuda:0')n_steps=100000batch_size=128test_batch_size=1000patience=200evaluation_step=50target='acc'model=MLPModel(n_steps=n_steps,batch_size=batch_size,test_batch_size=test_batch_size)history=model.fit(dataset_train=train_data,dataset_valid=valid_data,device=device,metric=target,patience=patience,evaluation_step=evaluation_step)#### Evaluate the trained modelmetric_value=model.test(test_data,target)🔧  Build a two-stage weak supervision pipelineimportloggingimporttorchfromwrench.datasetimportload_datasetfromwrench.loggingimportLoggingHandlerfromwrench.endmodelimportMLPModelfromwrench.labelmodelimportMajorityVoting#### Just some code to print debug information to stdoutlogging.basicConfig(format='%(asctime)s-%(message)s',datefmt='%Y-%m-%d%H:%M:%S',level=logging.INFO,handlers=[LoggingHandler()])logger=logging.getLogger(__name__)#### Load datasetdataset_home='../datasets'data='youtube'#### Extract data features using pre-trained BERT model and cache itextract_fn='bert'model_name='bert-base-cased'train_data,valid_data,test_data=load_dataset(dataset_home,data,extract_feature=True,extract_fn=extract_fn,cache_name=extract_fn,model_name=model_name)#### Generate soft training label via a label model#### The weak labels provided by supervision sources are alreadly encoded in dataset objectlabel_model=MajorityVoting()label_model.fit(train_data,valid_data)soft_label=label_model.predict_proba(train_data)#### Train a MLP classifier with soft labeldevice=torch.device('cuda:0')n_steps=100000batch_size=128test_batch_size=1000patience=200evaluation_step=50target='acc'model=MLPModel(n_steps=n_steps,batch_size=batch_size,test_batch_size=test_batch_size)history=model.fit(dataset_train=train_data,dataset_valid=valid_data,y_train=soft_label,device=device,metric=target,patience=patience,evaluation_step=evaluation_step)#### Evaluate the trained modelmetric_value=model.test(test_data,target)#### We can also train a MLP classifier with hard labelfromsnorkel.utilsimportprobs_to_predshard_label=probs_to_preds(soft_label)model=MLPModel(n_steps=n_steps,batch_size=batch_size,test_batch_size=test_batch_size)model.fit(dataset_train=train_data,dataset_valid=valid_data,y_train=hard_label,device=device,metric=target,patience=patience,evaluation_step=evaluation_step)🔧  Procedural labeling function generatorimportloggingimporttorchfromwrench.datasetimportload_datasetfromwrench.loggingimportLoggingHandlerfromwrench.syntheticimportConditionalIndependentGenerator,NGramLFGeneratorfromwrench.labelmodelimportFlyingSquid#### Just some code to print debug information to stdoutlogging.basicConfig(format='%(asctime)s-%(message)s',datefmt='%Y-%m-%d%H:%M:%S',level=logging.INFO,handlers=[LoggingHandler()])logger=logging.getLogger(__name__)#### Generate synthetic datasetgenerator=ConditionalIndependentGenerator(n_class=2,n_lfs=10,alpha=0.75,# mean accuracybeta=0.1,# mean propensityalpha_radius=0.2,# radius of accuracybeta_radius=0.1# radius of propensity)train_data=generator.generate_split('train',10000)valid_data=generator.generate_split('valid',1000)test_data=generator.generate_split('test',1000)#### Evaluate label model on synthetic datasetlabel_model=FlyingSquid()label_model.fit(dataset_train=train_data,dataset_valid=valid_data)target_value=label_model.test(test_data,metric_fn='auc')#### Load datasetdataset_home='../datasets'data='youtube'#### Load real-world datasettrain_data,valid_data,test_data=load_dataset(dataset_home,data,extract_feature=False)#### Generate procedural labeling functionsgenerator=NGramLFGenerator(dataset=train_data,min_acc_gain=0.1,min_support=0.01,ngram_range=(1,2))applier=generator.generate(mode='correlated',n_lfs=10)L_test=applier.apply(test_data)L_train=applier.apply(train_data)#### Evaluate label model on real-world dataset with semi-synthetic labeling functionslabel_model=FlyingSquid()label_model.fit(dataset_train=L_train,dataset_valid=valid_data)target_value=label_model.test(L_test,metric_fn='auc')🔧  ContactContact person: Jieyu Zhang,jieyuzhang97@gmail.comDon't hesitate to send us an e-mail if you have any question.We're also open to any collaboration!🔧  Contributing Dataset and ModelWe sincerely welcome any contribution to the datasets or models!