Package: cotk
Conversational Toolkitscotkis an open-source lightweight framework for model building and evaluation.
We provides standard dataset and evaluation suites in the domain of general language generation.
It easy to use and make you focus on designing your models!Features included:Light-weight, easy to start. Don't bother your way to construct models.Predefined standard datasets, in the domain of language modeling, dialog generation and more.Predefined evaluation suites, test your model with multiple metrics in several lines.A dashboard to show experiments, compare your and others' models fairly.Long-term maintenance and consistent development.This project is a part ofdialtk(Toolkits for Dialog System by Tsinghua University), you can followdialtkorcotkon our home page.Quick linksTutorial & DocumentsDashboardIndexInstallationRequirementsInstall from pipInstall from sourceQuick StartDataloaderMetricsPublish ExperimentsReproduce ExperimentsPredefined ModelsIssuesContributionsTeamLicenseInstallationRequirementspython 3numpy >= 1.13nltk >= 3.4tqdm >= 4.30checksumdir >= 1.1pytorch >= 1.0.0 (optional, accelerating the calculation of some metrics)transformers (optional, used for pretrained models)We support Unix, Windows, and macOS.Install from pipYou can simply get the latest stable version from pip usingpipinstallcotkInstall from source codeClone the cotk repositorygitclonehttps://github.com/thu-coai/cotk.gitInstall cotk via pipcdcotkpipinstall-e.Quick StartLet's skim through the whole package to find what you want.DataloaderLoad common used dataset and do preprocessing:Download online resources or import from local pathSplit training set, development set and test setConstruct vocabulary list>>>importcotk.dataloader>>># automatically download online resources>>>dataloader=cotk.dataloader.MSCOCO("resources://MSCOCO_small")>>># or download from a url>>>dl_url=cotk.dataloader.MSCOCO("http://cotk-data.s3-ap-northeast-1.amazonaws.com/mscoco_small.zip#MSCOCO")>>># or import from local file>>>dl_zip=cotk.dataloader.MSCOCO("./MSCOCO.zip#MSCOCO")>>>print("Dataset is split into:",dataloader.fields.keys())dict_keys(['train','dev','test'])Inspect vocabulary list>>>print("Vocabulary size:",dataloader.frequent_vocab_size)Vocabularysize:2597>>>print("First 10 tokens in vocabulary:",dataloader.frequent_vocab_list[:10])First10tokensinvocabulary:['<pad>','<unk>','<go>','<eos>','.','a','A','on','of','in']Convert between ids and strings>>>print("Convert string to ids",\...dataloader.convert_tokens_to_ids(["<go>","hello","world","<eos>"]))Convertstringtoids[2,6107,1875,3]>>>print("Convert ids to string",\...dataloader.convert_ids_to_tokens([2,1379,1897,3]))Convertidstostring['hello','world']Iterate over batches>>>fordataindataloader.get_batches("train",batch_size=1):...print(data){'sent':array([[2,181,13,26,145,177,8,22,12,5,1,1099,4,3]]),# <go> This is an old photo of people and a <unk> wagon.'sent_allvocabs':array([[2,181,13,26,145,177,8,22,12,5,3755,1099,4,3]]),# <go> This is an old photo of people and a horse-drawn wagon.'sent_length':array([14])}......Or usingwhile(another iteration method) if you like>>>dataloader.restart("train",batch_size=1):>>>whileTrue:...data=dataloader.get_next_batch("train")...ifdataisNone:break...print(data)note: If you want to know more aboutDataloader, please refer todocs of dataloader.MetricsWe found there are different versions of the same metric in different papers,
which leads tounfair comparison between models. For example, whether consideringunk, calculating the mean of NLL across sentences or tokens inperplexitymay introduce huge differences.We provide a unified implementation for metrics, wherehashvalueis provided for
checking whether the same data is used. The metric object receives mini-batches.>>>importcotk.metric>>>metric=cotk.metric.SelfBleuCorpusMetric(dataloader,gen_key="gen")>>>metric.forward({..."gen":...[[2,181,13,26,145,177,8,22,12,5,3755,1099,4,3],...[2,46,145,500,1764,207,11,5,93,7,31,4,3]]...})>>>print(metric.close()){'self-bleu':0.02253475750490193,'self-bleu hashvalue':'f7d75c0d0dbf53ffba4b845d1f61487fd2d6d3c0594b075c43111816c84c65fc'}You can merge multiple metrics together by cotk.metric.MetricChain.>>>metric=cotk.metric.MetricChain()>>>metric.add_metric(cotk.metric.SelfBleuCorpusMetric(dataloader,gen_key="gen"))>>>metric.add_metric(cotk.metric.FwBwBleuCorpusMetric(dataloader,reference_test_list=dataloader.get_all_batch('test')['sent_allvocabs'],gen_key="gen"))>>>metric.forward({..."gen":...[[2,181,13,26,145,177,8,22,12,5,3755,1099,4,3],...[2,46,145,500,1764,207,11,5,93,7,31,4,3]]...})>>>print(metric.close())100%|██████████|1000/1000[00:00<00:00,5281.95it/s]{'self-bleu':0.02253475750490193,'self-bleu hashvalue':'f7d75c0d0dbf53ffba4b845d1f61487fd2d6d3c0594b075c43111816c84c65fc','fw-bleu':0.28135593382545376,'bw-bleu':0.027021522872801896,'fw-bw-bleu':0.04930753293488745,'fw-bw-bleu hashvalue':'60a39f381e065e8df6fb5eb272984128c9aea7dee4ba50a43bfb768395a70762'}We also provide recommended metrics for selected dataloader.>>>metric=dataloader.get_inference_metric(gen_key="gen")>>>metric.forward({..."gen":...[[2,181,13,26,145,177,8,22,12,5,3755,1099,4,3],...[2,46,145,500,1764,207,11,5,93,7,31,4,3]]...})>>>print(metric.close())100%|██████████|1000/1000[00:00<00:00,4857.36it/s]100%|██████████|1250/1250[00:00<00:00,4689.29it/s]{'self-bleu':0.02253475750490193,'self-bleu hashvalue':'f7d75c0d0dbf53ffba4b845d1f61487fd2d6d3c0594b075c43111816c84c65fc','fw-bleu':0.3353037449663603,'bw-bleu':0.027327995838287513,'fw-bw-bleu':0.050537105917262654,'fw-bw-bleu hashvalue':'c254aa4008ae11b1bc4955e7cd1f7f3aad34b664178a585a218b1474970e3f23','gen':[['inside','is','an','elephant','shirt','of','people','and','a','grasslands','pulls','.'],['An','elephant','girls','baggage','sidewalk','with','a','clock','on','it','.']]}note: If you want to know more about metrics, please refer todocs of metrics.Predefined ModelsWe have provided some baselines for the classical tasks, seeModel Zooin docs for details.You can also usecotk download thu-coai/MODEL_NAME/masterto get the codes.IssuesYou are welcome to create an issue if you want to request a feature, report a bug or ask a general question.ContributionsWe welcome contributions from community.If you want to make a big change, we recommend first creating an issue with your design.Small contributions can be directly made by a pull request.If you like make contributions for our library, see issues to find what we need.Teamcotkis maintained and developed by Tsinghua university conversational AI group (THU-coai). Check ourmain pages(In Chinese).LicenseApache License 2.0