Omkar2118/datasets huggingface dataset:
Douban Conversation Corpus
Full name: Douban Conversation Corpus

We release Douban Conversation Corpus, comprising a training data set, a development set and a test set for retrieval based chatbot. The statistics of Douban Conversation Corpus are shown in the following table. 

|      |Train|Val| Test         | 
| ------------- |:-------------:|:-------------:|:-------------:|
| session-response pairs  | 1m|50k| 10k |
| Avg. positive response per session     | 1|1| 1.18    | 
| Fless Kappa | N\A|N\A|0.41      | 
| Min turn per session | 3|3| 3      | 
| Max ture per session | 98|91|45    | 
| Average turn per session | 6.69|6.75|5.95    | 
| Average Word per utterance | 18.56|18.50|20.74   | 


The test data contains 1000 dialogue context, and for each context we create 10 responses as candidates. We recruited three labelers to judge if a candidate is a proper response to the session. A proper response means the response can naturally reply to the message given the context. Each pair received three labels and the majority of the labels was taken as the final decision.

<br>
As far as we known, this is the first human-labeled test set for retrieval-based chatbots. The entire corpus link https://www.dropbox.com/s/90t0qtji9ow20ca/DoubanConversaionCorpus.zip?dl=0


## Data template
label \t conversation utterances (splited by \t) \t response

Introducing paper: Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots
E-commerce
Full name: E-commerce

We release E-commerce Dialogue Corpus, comprising a training data set, a development set and a test set for retrieval based chatbot. The statistics of E-commerical Conversation Corpus are shown in the following table. 

|      |Train|Val| Test         |
| ------------- |:-------------:|:-------------:|:-------------:|
| Session-response pairs  | 1m|10k| 10k |
| Avg. positive response per session|1|1|1|
| Min turn per session|3|3|3|
| Max ture per session|10|10|10|
| Average turn per session|5.51|5.48|5.64
| Average Word per utterance|7.02|6.99|7.11

The full corpus can be downloaded from https://drive.google.com/file/d/154J-neBo20ABtSmJDvm7DK0eTuieAuvw/view?usp=sharing.

Introducing paper: Modeling Multi-turn Conversation with Deep Utterance Aggregation
RRS
Full name: Restoration-200k for Response Selection

|           | Train | Validation | Test    | Ranking Test |
| --------- | ----- | ---------- | ------- | ------------ |
| size      | 0.4M  | 50K        | 5K      | 800          |
| pos:neg   | 1:1   | 1:9        | 1.2:8.8 | -            |
| avg turns | 5.0   | 5.0        | 5.0     | 5.0          |

Ranking test set contains the high-quality responses that selected by some baselines, and their correlation with the conversation context are carefully annotated by 8 professional annotators (the average annotation scores are saved for ranking). For ranking test set, the metrics should be NDCG@3 and NDCG@5, since the correlation scores are provided. More details are available in the Appendix of the paper.

Introducing paper: Exploring Dense Retrieval for Dialogue Response Selection
RRS Ranking Test
Full name: Restoration-200k for Response Selection with Ranking Test Set

|           | Train | Validation | Test    | Ranking Test |
| --------- | ----- | ---------- | ------- | ------------ |
| size      | 0.4M  | 50K        | 5K      | 800          |
| pos:neg   | 1:1   | 1:9        | 1.2:8.8 | -            |
| avg turns | 5.0   | 5.0        | 5.0     | 5.0          |

Ranking test set contains the high-quality responses that selected by some baselines, and their correlation with the conversation context are carefully annotated by 8 professional annotators (the average annotation scores are saved for ranking). For ranking test set, the metrics should be NDCG@3 and NDCG@5, since the correlation scores are provided. More details are available in the Appendix of the paper.

Introducing paper: Exploring Dense Retrieval for Dialogue Response Selection
MultiSV

**MultiSV** is a corpus designed for training and evaluating text-independent multi-channel speaker verification systems. It can be readily used also for experiments with dereverberation, denoising, and speech enhancement.

Introducing paper: MultiSV: Dataset for Far-Field Multi-Channel Speaker Verification