Repository: mem0ai/mem0
Pull request: Add dry_run to add()
## Description

Added **dry_run** method to `add` in Embedchain class.

Fixes #399 

## Type of change

Please delete options that are not relevant.

- [x] New feature (non-breaking change which adds functionality)
- [x] Documentation update

## How Has This Been Tested?

`poetry run pytest`

- [x] Unit Test

## Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
- [x] I have checked my code and corrected any misspellings


Problem statement:
feature request: `dry_run` for `add` method
### 🚀 The feature

A dry run method on add could show the chunks. This way you can make sure loader and chunker work as intended.

### Motivation, pitch

This way you have no cost when testing loaders.

### Alternatives

_No response_

### Additional context

It could be implemented in a similar way to `query` and `chat` dry_run.

Hints from issue discussion:

Commit message: return in dict format

Patch to review:
diff --git a/docs/advanced/interface_types.mdx b/docs/advanced/interface_types.mdx
index cc775a1528..41332a97b6 100644
--- a/docs/advanced/interface_types.mdx
+++ b/docs/advanced/interface_types.mdx
@@ -36,7 +36,7 @@ print(naval_chat_bot.chat("what did the author say about happiness?"))
 
 #### Dry Run
 
-Dry Run is an option in the `query` and `chat` methods that allows the user to not send their constructed prompt to the LLM, to save money. It's used for [testing](/advanced/testing#dry-run).
+Dry Run is an option in the `add`, `query` and `chat` methods that allows the user to displays the data chunks and their constructed prompt which is not send to the LLM, to save money. It's used for [testing](/advanced/testing#dry-run).
 
 
 ### Stream Response
diff --git a/docs/advanced/testing.mdx b/docs/advanced/testing.mdx
index ed1aaa97e1..7e9b38487a 100644
--- a/docs/advanced/testing.mdx
+++ b/docs/advanced/testing.mdx
@@ -6,11 +6,9 @@ title: '🧪 Testing'
 
 ### Dry Run
 
-Before you consume valueable tokens, you should make sure that the embedding you have done works and that it's receiving the correct document from the database.
+Before you consume valueable tokens, you should make sure that data chunks are properly created and the embedding you have done works and that it's receiving the correct document from the database.
 
-For this you can use the `dry_run` option in your `query` or `chat` method.
-
-Following the example above, add this to your script:
+- For `query` or `chat` method, you can add this to your script:
 
 ```python
 print(naval_chat_bot.query('Can you tell me who Naval Ravikant is?', dry_run=True))
@@ -26,4 +24,17 @@ A: Naval Ravikant is an Indian-American entrepreneur and investor.
 
 _The embedding is confirmed to work as expected. It returns the right document, even if the question is asked slightly different. No prompt tokens have been consumed._
 
-**The dry run will still consume tokens to embed your query, but it is only ~1/15 of the prompt.**
+The dry run will still consume tokens to embed your query, but it is only **~1/15 of the prompt.**
+
+
+- For `add` method, you can add this to your script:
+
+```python
+print(naval_chat_bot.add('https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf', dry_run=True))
+
+'''
+{'chunks': ['GETTING RICH IS NOT JUST ABOUT'], 'metadata': [{'source': 'C:\\Users\\Dev\\AppData\\Local\\Temp\\tmp7_cnh810\\tmp.pdf', 'page': 2, 'url': 'https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf', 'data_type': 'pdf_file'}, 'count': 9940, 'data_type': DataType.PDF_FILE]}
+
+# only one item for readability
+'''
+```
\ No newline at end of file
diff --git a/embedchain/apps/App.py b/embedchain/apps/App.py
index 693fd8042e..e825464c6c 100644
--- a/embedchain/apps/App.py
+++ b/embedchain/apps/App.py
@@ -15,7 +15,7 @@ class App(EmbedChain):
 
     adds(data_type, url): adds the data from the given URL to the vector db.
     query(query): finds answer to the given query using vector database and LLM.
-    dry_run(query): test your prompt without consuming tokens.
+    dry_run(query): test data chunks and your prompt without consuming tokens.
     """
 
     def __init__(self, config: AppConfig = None, system_prompt: Optional[str] = None):
diff --git a/embedchain/apps/CustomApp.py b/embedchain/apps/CustomApp.py
index ad20d1ebd8..bf92e70051 100644
--- a/embedchain/apps/CustomApp.py
+++ b/embedchain/apps/CustomApp.py
@@ -17,7 +17,7 @@ class CustomApp(EmbedChain):
 
     adds(data_type, url): adds the data from the given URL to the vector db.
     query(query): finds answer to the given query using vector database and LLM.
-    dry_run(query): test your prompt without consuming tokens.
+    dry_run(query): test data chunks your prompt without consuming tokens.
     """
 
     def __init__(self, config: CustomAppConfig = None, system_prompt: Optional[str] = None):
diff --git a/embedchain/embedchain.py b/embedchain/embedchain.py
index d379863b6b..8c1826a759 100644
--- a/embedchain/embedchain.py
+++ b/embedchain/embedchain.py
@@ -84,6 +84,7 @@ def add(
         data_type: Optional[DataType] = None,
         metadata: Optional[Dict] = None,
         config: Optional[AddConfig] = None,
+        dry_run=False,
     ):
         """
         Adds the data from the given URL to the vector db.
@@ -96,6 +97,7 @@ def add(
         :param metadata: Optional. Metadata associated with the data source.
         :param config: Optional. The `AddConfig` instance to use as configuration
         options.
+        :param dry_run: Optional. A dry run displays the chunks to ensure that the loader and chunker work as intended.
         :return: source_id, a md5-hash of the source, in hexadecimal representation.
         """
         if config is None:
@@ -131,11 +133,14 @@ def add(
         data_formatter = DataFormatter(data_type, config)
         self.user_asks.append([source, data_type.value, metadata])
         documents, _metadatas, _ids, new_chunks = self.load_and_embed(
-            data_formatter.loader, data_formatter.chunker, source, metadata, source_id
+            data_formatter.loader, data_formatter.chunker, source, metadata, source_id, dry_run
         )
         if data_type in {DataType.DOCS_SITE}:
             self.is_docs_site_instance = True
 
+        if dry_run:
+            return {"chunks": documents, "metadata": _metadatas, "count": len(documents), "type": data_type}
+
         # Send anonymous telemetry
         if self.config.collect_metrics:
             # it's quicker to check the variable twice than to count words when they won't be submitted.
@@ -169,7 +174,9 @@ def add_local(self, source, data_type=None, metadata=None, config: AddConfig = N
         )
         return self.add(source=source, data_type=data_type, metadata=metadata, config=config)
 
-    def load_and_embed(self, loader: BaseLoader, chunker: BaseChunker, src, metadata=None, source_id=None):
+    def load_and_embed(
+        self, loader: BaseLoader, chunker: BaseChunker, src, metadata=None, source_id=None, dry_run=None
+    ):
         """
         Loads the data from the given URL, chunks it, and adds it to database.
 
@@ -179,6 +186,7 @@ def load_and_embed(self, loader: BaseLoader, chunker: BaseChunker, src, metadata
         remote sources or local content for local loaders.
         :param metadata: Optional. Metadata associated with the data source.
         :param source_id: Hexadecimal hash of the source.
+        :param dry_run: Optional. A dry run returns chunks and doesn't update DB.
         :return: (List) documents (embedded text), (List) metadata, (list) ids, (int) number of chunks
         """
         embeddings_data = chunker.create_chunks(loader, src)
@@ -208,6 +216,9 @@ def load_and_embed(self, loader: BaseLoader, chunker: BaseChunker, src, metadata
             ids = list(data_dict.keys())
             documents, metadatas = zip(*data_dict.values())
 
+        if dry_run:
+            return list(documents), metadatas, ids, 0
+
         # Loop though all metadatas and add extras.
         new_metadatas = []
         for m in metadatas:


Code review comments:
diff_hunk: @@ -133,13 +133,13 @@ def add(
         data_formatter = DataFormatter(data_type, config)
         self.user_asks.append([source, data_type.value, metadata])
         documents, _metadatas, _ids, new_chunks = self.load_and_embed(
-            data_formatter.loader, data_formatter.chunker, source, metadata, source_id
+            data_formatter.loader, data_formatter.chunker, source, metadata, source_id, dry_run
path: embedchain/embedchain.py
text: @user1:
because metdatas is used now, you should remove the _underscore.

diff_hunk: @@ -33,6 +33,8 @@ The dry run will still consume tokens to embed your query, but it is only **~1/1
 print(naval_chat_bot.add('https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf', dry_run=True))
 
 '''
-['THE ALMANACK OF NAVAL RAVIKANT', 'GETTING RICH IS NOT JUST ABOUT LUCK; HAPPINESS IS NOT JUST A TRAIT WE ARE BORN WITH. These aspirations may seem out of reach, but building wealth and be - ing happy are skills we can learn. So what are these skills, and how do we learn them? What are the principles that should guide our efforts? What does progress really look like? Naval Ravikant is an entrepreneur, philosopher, and investor who has captivated the world with his principles for building wealth and creating long-term happiness. The Almanack of Naval Ravikant is a collection of Naval’s wisdom and experience from the last ten years, shared as a curation of his most insightful interviews and poignant reflections. This isn’t a how-to book, or a step-by-step gimmick. In - stead, through Naval’s own words, you will learn how to walk your own unique path toward a happier, wealthier life. This book has been created as a public service. It is available for free download in pdf and e-reader versions on Navalmanack.com. Naval is']
+{'chunks': ['GETTING RICH IS NOT JUST ABOUT'], 'metadata': [{'source': 'C:\\Users\\Dev\\AppData\\Local\\Temp\\tmp7_cnh810\\tmp.pdf', 'page': 2, 'url': 'https://navalmanack.s3.amazonaws.com/Eric-Jorgenson_The-Almanack-of-Naval-Ravikant_Final.pdf', 'data_type': 'pdf_file'}, 'count': 9940, 'data_type': DataType.PDF_FILE]}
+
+# only one item for readability
path: docs/advanced/testing.mdx
text: @user1:
I'm not trying to nitpick, but make it two or three to get the point across

@user1:
make the source something simpler like `~/doc.pdf`

@user1:
and try to prettify the dict

@user1:
thanks