Package: crslab
CRSLabPaper|Docs|中文版CRSLabis an open-source toolkit for building Conversational Recommender System (CRS). It is developed based on
Python and PyTorch. CRSLab has the following highlights:Comprehensive benchmark models and datasets: We have integrated commonly-used 6 datasets and 18 models, including graph neural network and pre-training models such as R-GCN, BERT and GPT-2. We have preprocessed these datasets to support these models, and release for downloading.Extensive and standard evaluation protocols: We support a series of widely-adopted evaluation protocols for testing and comparing different CRS.General and extensible structure: We design a general and extensible structure to unify various conversational recommendation datasets and models, in which we integrate various built-in interfaces and functions for quickly development.Easy to get started: We provide simple yet flexible configuration for new researchers to quickly start in our library.Human-machine interaction interfaces: We provide flexible human-machine interaction interfaces for researchers to conduct qualitative analysis.Figure 1: The overall framework of CRSLabInstallationQuick-StartModelsDatasetsPerformanceReleasesContributionsCitingTeamLicenseInstallationCRSLab works with the following operating systems：LinuxWindows 10macOS XCRSLab requires Python version 3.6 or later.CRSLab requires torch version 1.4.0 or later. If you want to use CRSLab with GPU, please ensure that CUDA or CUDAToolkit version is 9.2 or later. Please use the combinations shown in thisLinkto ensure the normal operation of PyTorch Geometric.Install PyTorchUse PyTorchLocally InstallationorPrevious Versions Installationcommands to install PyTorch. For example, on Linux and Windows 10:# CUDA 10.1pipinstalltorch==1.6.0+cu101torchvision==0.7.0+cu101-fhttps://download.pytorch.org/whl/torch_stable.html# CPU onlypipinstalltorch==1.6.0+cputorchvision==0.7.0+cpu-fhttps://download.pytorch.org/whl/torch_stable.htmlIf you want to use CRSLab with GPU, make sure the following command printsTrueafter installation:$python-c"import torch; print(torch.cuda.is_available())">>>TrueInstall PyTorch GeometricEnsure that at least PyTorch 1.4.0 is installed:$python-c"import torch; print(torch.__version__)">>>1.6.0Find the CUDA version PyTorch was installed with:$python-c"import torch; print(torch.version.cuda)">>>10.1Install the relevant packages:pipinstalltorch-scatter-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-sparse-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-cluster-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-spline-conv-fhttps://pytorch-geometric.com/whl/torch-${TORCH}+${CUDA}.html
pipinstalltorch-geometricwhere${CUDA}and${TORCH}should be replaced by your specific CUDA version (cpu,cu92,cu101,cu102,cu110) and PyTorch version (1.4.0,1.5.0,1.6.0,1.7.0) respectively. For example, for PyTorch 1.6.0 and CUDA 10.1, type:pipinstalltorch-scatter-fhttps://pytorch-geometric.com/whl/torch-1.6.0+cu101.html
pipinstalltorch-sparse-fhttps://pytorch-geometric.com/whl/torch-1.6.0+cu101.html
pipinstalltorch-cluster-fhttps://pytorch-geometric.com/whl/torch-1.6.0+cu101.html
pipinstalltorch-spline-conv-fhttps://pytorch-geometric.com/whl/torch-1.6.0+cu101.html
pipinstalltorch-geometricInstall CRSLabYou can install from pip:pipinstallcrslabOR install from source:gitclonehttps://github.com/RUCAIBox/CRSLab&&cdCRSLab
pipinstall-e.Quick-StartWith the source code, you can use the provided script for initial usage of our library with cpu by default:pythonrun_crslab.py--configconfig/crs/kgsf/redial.yamlThe system will complete the data preprocessing, and training, validation, testing of each model in turn. Finally it will get the evaluation results of specified models.If you want to save pre-processed datasets and training results of models, you can use the following command:pythonrun_crslab.py--configconfig/crs/kgsf/redial.yaml--save_data--save_systemIn summary, there are following arguments inrun_crslab.py:--configor-c: relative path for configuration file(yaml).--gpuor-g: specify GPU id(s) to use, we now support multiple GPUs. Defaults to CPU(-1).--save_dataor-sd: save pre-processed dataset.--restore_dataor-rd: restore pre-processed dataset from file.--save_systemor-ss: save trained system.--restore_systemor-rs: restore trained system from file.--debugor-d: use validation dataset to debug your system.--interactor-i: interact with your system instead of training.--tensorboardor-tb: enable tensorboard to monitor train performance.ModelsIn CRSLab, we unify the task description of conversational recommendation into three sub-tasks, namely recommendation (recommend user-preferred items), conversation (generate proper responses) and policy (select proper interactive action). The recommendation and conversation sub-tasks are the core of a CRS and have been studied in most of works. The policy sub-task is needed by recent works, by which the CRS can interact with users through purposeful strategy.
As the first release version, we have implemented 18 models in the four categories of CRS model, Recommendation model, Conversation model and Policy model.CategoryModelGraph Neural Network?Pre-training Model?CRS ModelReDialKBRDKGSFTG-ReDialINSPIRED×√√×××××√√Recommendation modelPopularityGRU4RecSASRecTextCNNR-GCNBERT××××√××××××√Conversation modelHERDTransformerGPT-2×××××√Policy modelPMIMGCGConv-BERTTopic-BERTProfile-BERT×××××××√√√Among them, the four CRS models integrate the recommendation model and the conversation model to improve each other, while others only specify an individual task.For Recommendation model and Conversation model, we have respectively implemented the following commonly-used automatic evaluation metrics:CategoryMetricsRecommendation MetricsHit@{1, 10, 50}, MRR@{1, 10, 50}, NDCG@{1, 10, 50}Conversation MetricsPPL, BLEU-{1, 2, 3, 4}, Embedding Average/Extreme/Greedy, Distinct-{1, 2, 3, 4}Policy MetricsAccuracy, Hit@{1,3,5}DatasetsWe have collected and preprocessed 6 commonly-used human-annotated datasets, and each dataset was matched with proper KGs as shown below:DatasetDialogsUtterancesDomainsTask DefinitionEntity KGWord KGReDial10,006182,150Movie--DBpediaConceptNetTG-ReDial10,000129,392MovieTopic GuideCN-DBpediaHowNetGoRecDial9,125170,904MovieAction ChoiceDBpediaConceptNetDuRecDial10,200156,000Movie, MusicGoal PlanCN-DBpediaHowNetINSPIRED1,00135,811MovieSocial StrategyDBpediaConceptNetOpenDialKG13,80291,209Movie, BookPath GenerateDBpediaConceptNetPerformanceWe have trained and test the integrated models on the TG-Redial dataset, which is split into training, validation and test sets using a ratio of 8:1:1. For each conversation, we start from the first utterance, and generate reply utterances or recommendations in turn by our model. We perform the evaluation on the three sub-tasks.Recommendation TaskModelHit@1Hit@10Hit@50MRR@1MRR@10MRR@50NDCG@1NDCG@10NDCG@50SASRec0.0004460.001340.01600.0004460.0005760.001140.0004450.000750.00380TextCNN0.002670.01030.02360.002670.004340.004930.002670.005700.00860BERT0.007220.004900.02810.007220.01060.01240.004900.01470.0239KBRD0.004010.02540.05880.004010.008910.01030.004010.01270.0198KGSF0.005350.02850.07710.005350.01140.01350.005350.01540.0259TG-ReDial0.007930.02510.05240.007930.01220.01340.007930.01520.0211Conversation TaskModelBLEU@1BLEU@2BLEU@3BLEU@4Dist@1Dist@2Dist@3Dist@4AverageExtremeGreedyPPLHERD0.1200.01410.001360.0003500.1810.3690.8471.300.6970.3820.639472Transformer0.2660.04400.01450.006510.3240.8372.023.060.8790.4380.68030.9GPT20.08580.01190.003770.01102.354.628.8412.50.7630.2970.5839.26KBRD0.2670.04580.01340.005790.4691.503.404.900.8630.3980.71052.5KGSF0.3830.1150.04440.02000.3400.9103.506.200.8880.4770.76750.1TG-ReDial0.1250.02040.003540.0008030.8811.757.0012.00.8100.3320.5987.41Policy TaskModelHit@1Hit@10Hit@50MRR@1MRR@10MRR@50NDCG@1NDCG@10NDCG@50MGCG0.5910.8180.8830.5910.6800.6830.5910.7120.729Conv-BERT0.5970.8140.8810.5970.6840.6870.5970.7160.731Topic-BERT0.5980.8280.8850.5980.6900.6930.5980.7240.737TG-ReDial0.6000.8300.8930.6000.6930.6960.6000.7270.741The above results were obtained from our CRSLab in preliminary experiments. However, these algorithms were implemented and tuned based on our understanding and experiences, which may not achieve their optimal performance. If you could yield a better result for some specific algorithm, please kindly let us know. We will update this table after the results are verified.ReleasesReleasesDateFeaturesv0.1.11 / 4 / 2021Basic CRSLabv0.1.23 / 28 / 2021CRSLabContributionsPlease let us know if you encounter a bug or have any suggestions byfiling an issue.We welcome all contributions from bug fixes to new features and extensions.We expect all contributions discussed in the issue tracker and going through PRs.We thank the nice contributions through PRs from@shubaoyu,@ToheartZhang.CitingIf you find CRSLab useful for your research or development, please cite ourPaper:@article{crslab,
    title={CRSLab: An Open-Source Toolkit for Building Conversational Recommender System},
    author={Kun Zhou, Xiaolei Wang, Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li, Ji-Rong Wen},
    year={2021},
    journal={arXiv preprint arXiv:2101.00939}
}TeamCRSLabwas developed and maintained byAI Boxgroup in RUC.LicenseCRSLabusesMIT License.