Repository: run-llama/llama_index
Pull request: Implement mongodb kvstore put all method
# Description

When I assign batch_size greater than 1 for MongoDBstore, it fails because put_all in MongoDBKVStore is not implemented. 

Fixes https://github.com/run-llama/llama_index/issues/9998

## Type of Change

Please delete options that are not relevant.

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] This change requires a documentation update

# How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

- [ ] Added new unit/integration tests
- [ ] Added new notebook (that tests end-to-end)
- [x] I stared at the code and made sure it makes sense

# Suggested Checklist:

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] I have added Google Colab support for the newly added notebooks.
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I ran `make format; make lint` to appease the lint gods


Problem statement:
[Feature Request]: Implement put_all for MongoDBKVStore
### Feature Description

Hi folks, I was wondering if i'm just using this incorrectly, but when I assign batch_size greater than 1 for MongoDBstore, it fails because put_all in MongoDBKVStore is not implemented. At the moment, i've had to create a separate MongoDBStore/KVStore with put_all implemented to work around this issue, but it would be great if put_all was natively supported out of the box.

The fix for me was to simply just define it as:
```
    def put_all(self, kv_pairs: List[Tuple[str, dict]], collection: str = DEFAULT_COLLECTION) -> None:
        formatted_documents = [{'_id': key, **value} for key, value in kv_pairs]
        self._db[collection].insert_many(formatted_documents)
```

### Reason

Not sure what's stopping Llamaindex from natively supporting this feature. 

### Value of Feature

It will allow bulk inserts into MongoDB instead of using individual put calls. This will reduce latency for when we need to upload large quantities of nodes at once.

Hints from issue discussion:
_🤖_
<!-- Greeting -->
Hello @kimchinosys! 

I'm [Dosu](https://dosu.dev), a bot here to help you navigate through LlamaIndex issues and contributions. While we're waiting for a human maintainer, I'll do my best to assist you. 

I've seen your feature request and I'm currently taking a look at it. Hang tight, I'll get back to you shortly with a more detailed response.



@kimchinosys we are adding batch and async one at a time (its very new). Would be extremely appreciated if you submitted a PR for this 
Sure, do i need additional permissions to make the PR
@kimchinosys nope, anyone can create a fork (which makes a copy of the repo on your github account), and then you can contribute any changes from your fork to the repo in a PR 👍🏻

Commit message: Implement mongodb kvstore put all method

Patch to review:
diff --git a/llama_index/storage/kvstore/mongodb_kvstore.py b/llama_index/storage/kvstore/mongodb_kvstore.py
index 045e30cdb8..b60a1efd5f 100644
--- a/llama_index/storage/kvstore/mongodb_kvstore.py
+++ b/llama_index/storage/kvstore/mongodb_kvstore.py
@@ -1,6 +1,6 @@
-from typing import Any, Dict, Optional, cast
+from typing import Any, Dict, List, Optional, cast, Tuple
 
-from llama_index.storage.kvstore.types import DEFAULT_COLLECTION, BaseKVStore
+from llama_index.storage.kvstore.types import DEFAULT_BATCH_SIZE, DEFAULT_COLLECTION, BaseKVStore
 
 IMPORT_ERROR_MSG = "`pymongo` package not found, please run `pip install pymongo`"
 
@@ -130,6 +130,13 @@ async def aput(
 
         """
         raise NotImplementedError
+    
+    def put_all(self, kv_pairs: List[Tuple[str, dict]], collection: str = DEFAULT_COLLECTION, batch_size: int = DEFAULT_BATCH_SIZE) -> None:
+        # Prepare documents with '_id' set to the key for batch insertion
+        docs = [{'_id': key, **value} for key, value in kv_pairs]
+        # Insert documents in batches
+        for batch in (docs[i:i + batch_size] for i in range(0, len(docs), batch_size)):
+            self._db[collection].insert_many(batch)
 
     def get(self, key: str, collection: str = DEFAULT_COLLECTION) -> Optional[dict]:
         """Get a value from the store.


Code review comments:
diff_hunk: @@ -130,6 +130,13 @@ async def aput(
 
         """
         raise NotImplementedError
+    
+    def put_all(self, kv_pairs: List[Tuple[str, dict]], collection: str = DEFAULT_COLLECTION, batch_size: int = DEFAULT_BATCH_SIZE) -> None:
path: llama_index/storage/kvstore/mongodb_kvstore.py
text: @user1:
can you add async func for this as well

@author:
Unfortunately (at the moment) i don't have the time to properly test the async func for put_all if i implemented it. Can we merge this individually?