Deprecate search API
State: open | Author: mariosasko (COLLABORATOR)

The Search API only supports Faiss and ElasticSearch as vector stores, is somewhat difficult to maintain (e.g., it still doesn't support ElasticSeach 8.0, difficult testing, ...), does not have the best design (adds a bunch of methods to the `Dataset` class that are only useful after creating an index), the usage doesn't seem to be significant and is not integrated with the Hub. Since we have no plans/bandwidth to improve it and better alternatives such as `langchain` and `docarray` exist, I think it should be deprecated (and eventually removed).

If we decide to deprecate/remove it, the following usage instances need to be addressed:
* [Course](https://github.com/huggingface/course/blob/0018bb434204d9750a03592cb0d4e846093218d8/chapters/en/chapter5/6.mdx#L342 ) and [Blog](https://github.com/huggingface/blog/blob/4897c6f73d4492a0955ade503281711d01840e09/image-search-datasets.md?plain=1#L252) - calling the FAISS API directly should be OK in these instances as it's pretty simple to use for basic scenarios. Alternatively, we can use `langchain`, but this adds an extra dependency
*  [Transformers](https://github.com/huggingface/transformers/blob/50726f9ea7afc6113da617f8f4ca1ab264a5e28a/src/transformers/models/rag/retrieval_rag.py#L183) - we can use the FAISS API directly and store the index as a separate attribute (and instead of building the `wiki_dpr` index each time the dataset is generated, we can generate it once and push it to the Hub repo, and then read it from there

cc @huggingface/datasets @LysandreJik for the opinion

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005746 / 0.011353 (-0.005607) | 0.003461 / 0.011008 (-0.007548) | 0.078672 / 0.038508 (0.040164) | 0.056800 / 0.023109 (0.033691) | 0.312853 / 0.275898 (0.036955) | 0.346715 / 0.323480 (0.023235) | 0.004516 / 0.007986 (-0.003469) | 0.002872 / 0.004328 (-0.001457) | 0.061264 / 0.004250 (0.057013) | 0.046606 / 0.037052 (0.009553) | 0.320080 / 0.258489 (0.061591) | 0.350390 / 0.293841 (0.056550) | 0.026445 / 0.128546 (-0.102101) | 0.007710 / 0.075646 (-0.067936) | 0.259519 / 0.419271 (-0.159752) | 0.043935 / 0.043533 (0.000402) | 0.320015 / 0.255139 (0.064876) | 0.339799 / 0.283200 (0.056599) | 0.018638 / 0.141683 (-0.123044) | 1.463393 / 1.452155 (0.011239) | 1.496977 / 1.492716 (0.004261) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.185175 / 0.018006 (0.167168) | 0.420734 / 0.000490 (0.420245) | 0.002569 / 0.000200 (0.002369) | 0.000067 / 0.000054 (0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022335 / 0.037411 (-0.015077) | 0.071686 / 0.014526 (0.057161) | 0.079906 / 0.176557 (-0.096650) | 0.140386 / 0.737135 (-0.596749) | 0.079712 / 0.296338 (-0.216627) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.392643 / 0.215209 (0.177434) | 3.917934 / 2.077655 (1.840279) | 1.906808 / 1.504120 (0.402688) | 1.729564 / 1.541195 (0.188369) | 1.751533 / 1.468490 (0.283043) | 0.496810 / 4.584777 (-4.087967) | 3.047405 / 3.745712 (-0.698307) | 4.361766 / 5.269862 (-0.908095) | 2.660845 / 4.565676 (-1.904832) | 0.056951 / 0.424275 (-0.367324) | 0.006277 / 0.007607 (-0.001330) | 0.466357 / 0.226044 (0.240312) | 4.660457 / 2.268929 (2.391529) | 2.328590 / 55.444624 (-53.116034) | 1.986140 / 6.876477 (-4.890337) | 2.096182 / 2.142072 (-0.045891) | 0.581685 / 4.805227 (-4.223542) | 0.123643 / 6.500664 (-6.377021) | 0.060286 / 0.075469 (-0.015183) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.237024 / 1.841788 (-0.604763) | 17.778533 / 8.074308 (9.704225) | 13.202205 / 10.191392 (3.010813) | 0.141301 / 0.680424 (-0.539123) | 0.016453 / 0.534201 (-0.517748) | 0.329173 / 0.579283 (-0.250110) | 0.349945 / 0.434364 (-0.084419) | 0.375319 / 0.540337 (-0.165018) | 0.530394 / 1.386936 (-0.856542) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005863 / 0.011353 (-0.005489) | 0.003578 / 0.011008 (-0.007430) | 0.062719 / 0.038508 (0.024211) | 0.056192 / 0.023109 (0.033082) | 0.422812 / 0.275898 (0.146914) | 0.454316 / 0.323480 (0.130836) | 0.004446 / 0.007986 (-0.003540) | 0.002808 / 0.004328 (-0.001521) | 0.062819 / 0.004250 (0.058569) | 0.046243 / 0.037052 (0.009190) | 0.445858 / 0.258489 (0.187369) | 0.463750 / 0.293841 (0.169909) | 0.027504 / 0.128546 (-0.101042) | 0.007897 / 0.075646 (-0.067749) | 0.068248 / 0.419271 (-0.351024) | 0.041921 / 0.043533 (-0.001612) | 0.413314 / 0.255139 (0.158175) | 0.441619 / 0.283200 (0.158419) | 0.019246 / 0.141683 (-0.122437) | 1.457069 / 1.452155 (0.004914) | 1.524168 / 1.492716 (0.031452) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237785 / 0.018006 (0.219779) | 0.418455 / 0.000490 (0.417965) | 0.002301 / 0.000200 (0.002101) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025630 / 0.037411 (-0.011781) | 0.076673 / 0.014526 (0.062147) | 0.084877 / 0.176557 (-0.091680) | 0.137528 / 0.737135 (-0.599607) | 0.085261 / 0.296338 (-0.211077) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419040 / 0.215209 (0.203831) | 4.183022 / 2.077655 (2.105368) | 2.157852 / 1.504120 (0.653732) | 1.966177 / 1.541195 (0.424982) | 2.019612 / 1.468490 (0.551122) | 0.497415 / 4.584777 (-4.087362) | 3.102873 / 3.745712 (-0.642839) | 4.526336 / 5.269862 (-0.743525) | 2.991503 / 4.565676 (-1.574174) | 0.057235 / 0.424275 (-0.367040) | 0.006735 / 0.007607 (-0.000872) | 0.498255 / 0.226044 (0.272211) | 4.957364 / 2.268929 (2.688435) | 2.632643 / 55.444624 (-52.811981) | 2.249788 / 6.876477 (-4.626688) | 2.289134 / 2.142072 (0.147062) | 0.583581 / 4.805227 (-4.221646) | 0.126046 / 6.500664 (-6.374618) | 0.062966 / 0.075469 (-0.012504) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.295215 / 1.841788 (-0.546573) | 18.554020 / 8.074308 (10.479711) | 13.683273 / 10.191392 (3.491881) | 0.132266 / 0.680424 (-0.548158) | 0.016376 / 0.534201 (-0.517825) | 0.334495 / 0.579283 (-0.244788) | 0.347106 / 0.434364 (-0.087258) | 0.387531 / 0.540337 (-0.152806) | 0.525745 / 1.386936 (-0.861191) |

</details>
</details>

![](https://cml.dev/watermark.png#f22aa0bf179602e4bf3f44a9de5d180579bb377e "CML watermark")


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6036). All of your documentation changes will be reflected on that endpoint.

I don't think `transformers` should have any dataset indexing code. So before deprecating I'd be in favor of finding a suitable replacement. Not sure about the stats of the RAG model that uses `datasets` indexing though

The RAG downloads stats are decent (over 20k downloads last month).

I think it's suboptimal to maintain an API that only a single model uses. One option is to put this code into a separate lib. However, `langchain` and `docarray` already provide a unified interface to vector stores, so I don't see this as an impactful project. Considering how specific this model is, I think we should go with the simplest solution and combine an index with a dataset in Transformers (this wouldn't require too much code).

What about migrating to the [datasets-server](https://github.com/huggingface/datasets-server) search feature instead? Would make more sense from a product perspective 

I don't think it's a good idea:
- using datasets-server would require to upload the data and to not control the indexing, whereas the current feature is about using a local index that you control
- faiss indexes are vector indexes that are not supported by datasets-server, and they are also very customised. For instance RAG uses DPR embeddings and cosine similarity
- FTS is only done for the first 5GB of data for now in datasets-server

I think a better option would be to integrate with open source search tools such as docarray.
And if we want to make the datasets-server search available in python we can build an integration in docarray and/or in huggingface_hub.

`llama_index` is another popular tool in this space.

@lhoestq 
> I think a better option would be to integrate with open source search tools such as docarray.
And if we want to make the datasets-server search available in python we can build an integration in docarray and/or in huggingface_hub.

I don't think these integrations would be popular unless we integrate them with the Hub "UI-wise" (e.g., through a widget), so they can wait IMO. Also, FAISS supports `fsspec` already with the callback reader/writer, so this doesn't require a specific integration. 

After discussing it a bit with @lhoestq, do we need to deprecate the search API? While I understand it's imperfect, it looks like this will result in significant work to update it everywhere, so I'd favor keeping it until there's an obviously better alternative; this way we can focus on different things in the meantime.

FAISS/ES are simple to use (probably the main reason why they are so popular), so creating "better alternatives" is not easy - they usually add more complexity (as is the case here, `langchain`, etc.)

So, instead of waiting for better alternatives, IMO it makes more sense to wait for the RAG model to be deprecated in Transformers (less than 1,000 cumulated downloads over all checkpoints in the past 30 days) before deprecating this API here.

In the meantime, we should make it clear that the vector search API is in maintenance mode (no new features, etc.).

How does that sound?