Repository: mem0ai/mem0
Pull request: [Feature] GMAIL Loader
## Description

This adds the ability to load gmail data into the app.

Fixes # ([338](https://github.com/embedchain/embedchain/issues/338))

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] Refactor (does not change functionality, e.g. code style improvements, linting)
- [x] Documentation update

## How Has This Been Tested?

Wrote a unit test and tested using a test script.

- [x] Unit Test
- [x] Test Script (please provide)

```
import os

from embedchain.apps.app import App
from embedchain.models.data_type import DataType

os.environ["OPENAI_API_KEY"] = "sk-xxx"

app = App()

gmail_query = "from: *@slack.com label:inbox"
app.add(gmail_query, data_type=DataType.GMAIL)

response = app.query("what is the email I received the slack invite from on deven@embedchain.ai?")
# Answer: The email you received the Slack invite from on deven@embedchain.ai is from Taranjeet (taranjeet@embedchain.ai).
```

## Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream modules
- [x] I have checked my code and corrected any misspellings

## Maintainer Checklist

- [x] closes #338 
- [x] Made sure Checks passed


Problem statement:
add support for email request
### 🚀 The feature

For emails, you should figure out how to import and understand .mbox files. Gmail allows you to export all your emails in one go and exports them in .mbox format

### Motivation, pitch

- opened on behalf of user Kushal from whatsapp

### Alternatives

_No response_

### Additional context

_No response_

Hints from issue discussion:

Commit message: code clean up

Patch to review:
diff --git a/.gitignore b/.gitignore
index b6df6ac072..b0b048ace7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -176,3 +176,8 @@ notebooks/*.yaml
 .ipynb_checkpoints/
 
 !configs/*.yaml
+
+
+# Google API
+credentials.json
+token.json
\ No newline at end of file
diff --git a/docs/data-sources/gmail.mdx b/docs/data-sources/gmail.mdx
new file mode 100644
index 0000000000..c8c1828790
--- /dev/null
+++ b/docs/data-sources/gmail.mdx
@@ -0,0 +1,35 @@
+---
+title: '📬 GMAIL'
+---
+
+To use GMAILLoader you must install the extra dependencies with `pip install --upgrade embedchain[dataloaders]`.
+
+The `source` must be a valid GMAIL search query, you can refer `https://support.google.com/mail/answer/7190?hl=en` to build a query.
+
+To load GMAIL messages, you MUST use the data_type as `gmail`. Otherwise the source will be detected as simple `text`.
+
+To use this you need `credentials.json` in your current directory. Follow these steps to get the credentials
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/apis/credentials).
+2. Create a project if you don't have one already.
+3. Create an `OAuth Consent Screen` in the project. You may need to select the `external` option.
+4. Make sure the consent screen is published.
+5. Enable the [Gmail API](https://console.cloud.google.com/apis/api/gmail.googleapis.com)
+6. Create credentials from the `Credentials` tab.
+7. Select the type `OAuth Client ID`.
+8. Choose the application type `Web application`. As a name you can choose `embedchain` or any other name as per your use case.
+9. Add an authorized redirect URI for `http://localhost:8080/`.
+10. You can leave everything else at default, finish the creation.
+11. When you are done, a modal opens where you can download the details in `json` format.
+12. Put the `.json` file in your current directory and rename it to `credentials.json`
+
+```python
+import os
+from embedchain.apps.app import App
+from embedchain.models.data_type import DataType
+app = App()
+
+query = "to: me label:inbox"
+app.add(query, data_type=DataType.GMAIL)
+app.query("Summarize my email conversations")
+```
\ No newline at end of file
diff --git a/docs/data-sources/notion.mdx b/docs/data-sources/notion.mdx
index faa52de07d..d6c616df8b 100644
--- a/docs/data-sources/notion.mdx
+++ b/docs/data-sources/notion.mdx
@@ -2,7 +2,7 @@
 title: '📓 Notion'
 ---
 
-To use notion you must install the extra dependencies with `pip install --upgrade embedchain[notion]`.
+To use notion you must install the extra dependencies with `pip install --upgrade embedchain[community]`.
 
 To load a notion page, use the data_type as `notion`. Since it is hard to automatically detect, it is advised to specify the `data_type` when adding a notion document.
 The next argument must **end** with the `notion page id`. The id is a 32-character string. Eg:
diff --git a/docs/data-sources/overview.mdx b/docs/data-sources/overview.mdx
index f0b9814296..1ba742fb42 100644
--- a/docs/data-sources/overview.mdx
+++ b/docs/data-sources/overview.mdx
@@ -19,6 +19,7 @@ Embedchain comes with built-in support for various data sources. We handle the c
   <Card title="🧾 xml" href="/data-sources/xml"></Card>
   <Card title="🙌 OpenAPI" href="/data-sources/openapi"></Card>
   <Card title="🎥📺 youtube video" href="/data-sources/youtube-video"></Card>
+  <Card title="📬 GMAIL" href="/data-sources/gmail"></Card>
 </CardGroup>
 
 <br/ >
diff --git a/embedchain/chunkers/gmail.py b/embedchain/chunkers/gmail.py
new file mode 100644
index 0000000000..edad45edf0
--- /dev/null
+++ b/embedchain/chunkers/gmail.py
@@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class GMAILChunker(BaseChunker):
+    """Chunker for gmail."""
+
+    def __init__(self, config: Optional[ChunkerConfig] = None):
+        if config is None:
+            config = ChunkerConfig(chunk_size=1000, chunk_overlap=0, length_function=len)
+        text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=config.chunk_size,
+            chunk_overlap=config.chunk_overlap,
+            length_function=config.length_function,
+        )
+        super().__init__(text_splitter)
diff --git a/embedchain/data_formatter/data_formatter.py b/embedchain/data_formatter/data_formatter.py
index d90064217c..56f80a662d 100644
--- a/embedchain/data_formatter/data_formatter.py
+++ b/embedchain/data_formatter/data_formatter.py
@@ -1,6 +1,7 @@
 from embedchain.chunkers.base_chunker import BaseChunker
 from embedchain.chunkers.docs_site import DocsSiteChunker
 from embedchain.chunkers.docx_file import DocxFileChunker
+from embedchain.chunkers.gmail import GMAILChunker
 from embedchain.chunkers.images import ImagesChunker
 from embedchain.chunkers.json import JSONChunker
 from embedchain.chunkers.mdx import MdxChunker
@@ -22,6 +23,7 @@
 from embedchain.loaders.csv import CsvLoader
 from embedchain.loaders.docs_site_loader import DocsSiteLoader
 from embedchain.loaders.docx_file import DocxFileLoader
+from embedchain.loaders.gmail import GMAILLoader
 from embedchain.loaders.images import ImagesLoader
 from embedchain.loaders.json import JSONLoader
 from embedchain.loaders.local_qna_pair import LocalQnaPairLoader
@@ -84,6 +86,7 @@ def _get_loader(self, data_type: DataType, config: LoaderConfig) -> BaseLoader:
             DataType.UNSTRUCTURED: UnstructuredLoader,
             DataType.JSON: JSONLoader,
             DataType.OPENAPI: OpenAPILoader,
+            DataType.GMAIL: GMAILLoader,
         }
         lazy_loaders = {DataType.NOTION}
         if data_type in loaders:
@@ -128,6 +131,7 @@ def _get_chunker(self, data_type: DataType, config: ChunkerConfig) -> BaseChunke
             DataType.UNSTRUCTURED: UnstructuredFileChunker,
             DataType.JSON: JSONChunker,
             DataType.OPENAPI: OpenAPIChunker,
+            DataType.GMAIL: GMAILChunker,
         }
         if data_type in chunker_classes:
             chunker_class: type = chunker_classes[data_type]
diff --git a/embedchain/loaders/gmail.py b/embedchain/loaders/gmail.py
new file mode 100644
index 0000000000..54250829bc
--- /dev/null
+++ b/embedchain/loaders/gmail.py
@@ -0,0 +1,124 @@
+import hashlib
+import logging
+import os
+import quopri
+from textwrap import dedent
+
+from bs4 import BeautifulSoup
+
+try:
+    from llama_index import download_loader
+except ImportError:
+    raise ImportError("Notion requires extra dependencies. Install with `pip install embedchain[community]`") from None
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+def get_header(text: str, header: str) -> str:
+    start_string_position = text.find(header)
+    pos_start = text.find(":", start_string_position) + 1
+    pos_end = text.find("\n", pos_start)
+    header = text[pos_start:pos_end]
+    return header.strip()
+
+
+class GMAILLoader(BaseLoader):
+    def load_data(self, query):
+        """Load data from gmail."""
+        if not os.path.isfile("credentials.json"):
+            raise FileNotFoundError(
+                "You must download the valid credentials file from your google \
+                dev account. Refer this `https://cloud.google.com/docs/authentication/api-keys`"
+            )
+
+        GmailReader = download_loader("GmailReader")
+        loader = GmailReader(query=query, service=None, results_per_page=20)
+        documents = loader.load_data()
+        logging.info(f"GMAIL Loader: {len(documents)} mails found for query- {query}")
+
+        data = []
+        data_contents = []
+        logging.info(f"Gmail Loader: {len(documents)} mails found")
+        for document in documents:
+            original_size = len(document.text)
+
+            snippet = document.metadata.get("snippet")
+            meta_data = {
+                "url": document.metadata.get("id"),
+                "date": get_header(document.text, "Date"),
+                "subject": get_header(document.text, "Subject"),
+                "from": get_header(document.text, "From"),
+                "to": get_header(document.text, "To"),
+                "search_query": query,
+            }
+
+            # Decode
+            decoded_bytes = quopri.decodestring(document.text)
+            decoded_str = decoded_bytes.decode("utf-8", errors="replace")
+
+            # Slice
+            mail_start = decoded_str.find("<!DOCTYPE")
+            email_data = decoded_str[mail_start:]
+
+            # Web Page HTML Processing
+            soup = BeautifulSoup(email_data, "html.parser")
+
+            tags_to_exclude = [
+                "nav",
+                "aside",
+                "form",
+                "header",
+                "noscript",
+                "svg",
+                "canvas",
+                "footer",
+                "script",
+                "style",
+            ]
+
+            for tag in soup(tags_to_exclude):
+                tag.decompose()
+
+            ids_to_exclude = ["sidebar", "main-navigation", "menu-main-menu"]
+            for id in ids_to_exclude:
+                tags = soup.find_all(id=id)
+                for tag in tags:
+                    tag.decompose()
+
+            classes_to_exclude = [
+                "elementor-location-header",
+                "navbar-header",
+                "nav",
+                "header-sidebar-wrapper",
+                "blog-sidebar-wrapper",
+                "related-posts",
+            ]
+
+            for class_name in classes_to_exclude:
+                tags = soup.find_all(class_=class_name)
+                for tag in tags:
+                    tag.decompose()
+
+            content = soup.get_text()
+            content = clean_string(content)
+
+            cleaned_size = len(content)
+            if original_size != 0:
+                logging.info(
+                    f"[{id}] Cleaned page size: {cleaned_size} characters, down from {original_size} (shrunk: {original_size-cleaned_size} chars, {round((1-(cleaned_size/original_size)) * 100, 2)}%)"  # noqa:E501
+                )
+
+            result = f"""
+            email from '{meta_data.get('from')}' to '{meta_data.get('to')}'
+            subject: {meta_data.get('subject')}
+            date: {meta_data.get('date')}
+            preview: {snippet}
+            content: f{content}
+            """
+            data_content = dedent(result)
+            data.append({"content": data_content, "meta_data": meta_data})
+            data_contents.append(data_content)
+        doc_id = hashlib.sha256((query + ", ".join(data_contents)).encode()).hexdigest()
+        response_data = {"doc_id": doc_id, "data": data}
+        return response_data
diff --git a/embedchain/models/data_type.py b/embedchain/models/data_type.py
index dd256fb6d8..31f07cdffa 100644
--- a/embedchain/models/data_type.py
+++ b/embedchain/models/data_type.py
@@ -28,6 +28,7 @@ class IndirectDataType(Enum):
     UNSTRUCTURED = "unstructured"
     JSON = "json"
     OPENAPI = "openapi"
+    GMAIL = "gmail"
 
 
 class SpecialDataType(Enum):
@@ -55,3 +56,4 @@ class DataType(Enum):
     UNSTRUCTURED = IndirectDataType.UNSTRUCTURED.value
     JSON = IndirectDataType.JSON.value
     OPENAPI = IndirectDataType.OPENAPI.value
+    GMAIL = IndirectDataType.GMAIL.value
diff --git a/embedchain/utils.py b/embedchain/utils.py
index 9b3e1629db..b2b30cf2f3 100644
--- a/embedchain/utils.py
+++ b/embedchain/utils.py
@@ -259,6 +259,8 @@ def is_openapi_yaml(yaml_content):
     else:
         # Source is not a URL.
 
+        # TODO: check if source is gmail query
+
         # Use text as final fallback.
         logging.debug(f"Source of `{formatted_source}` detected as `text`.")
         return DataType.TEXT
diff --git a/pyproject.toml b/pyproject.toml
index 5961e708c4..5515e83701 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -168,6 +168,13 @@ dataloaders=[
     "pytube",
     "sentence-transformers",
     "unstructured",
+    "jq",
+    "requests",
+    "google-api-python-client",
+    "google-auth",
+    "google-auth-oauthlib",
+    "google-auth-httplib2",
+    "google-api-core",
 ]
 vertexai = ["google-cloud-aiplatform"]
 llama2 = ["replicate"]
diff --git a/tests/chunkers/test_chunkers.py b/tests/chunkers/test_chunkers.py
index b7ee136e62..5518c89279 100644
--- a/tests/chunkers/test_chunkers.py
+++ b/tests/chunkers/test_chunkers.py
@@ -1,5 +1,6 @@
 from embedchain.chunkers.docs_site import DocsSiteChunker
 from embedchain.chunkers.docx_file import DocxFileChunker
+from embedchain.chunkers.gmail import GMAILChunker
 from embedchain.chunkers.json import JSONChunker
 from embedchain.chunkers.mdx import MdxChunker
 from embedchain.chunkers.notion import NotionChunker
@@ -31,6 +32,7 @@
     YoutubeVideoChunker: {"chunk_size": 2000, "chunk_overlap": 0, "length_function": len},
     JSONChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
     OpenAPIChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
+    GMAILChunker: {"chunk_size": 1000, "chunk_overlap": 0, "length_function": len},
 }
 
 
diff --git a/tests/loaders/test_gmail.py b/tests/loaders/test_gmail.py
new file mode 100644
index 0000000000..f111c1d4af
--- /dev/null
+++ b/tests/loaders/test_gmail.py
@@ -0,0 +1,49 @@
+import pytest
+from llama_index.readers.schema.base import Document
+
+from embedchain.loaders.gmail import GMAILLoader
+
+
+@pytest.fixture
+def mock_download_loader(mocker):
+    return mocker.patch("embedchain.loaders.gmail.download_loader")
+
+
+@pytest.fixture
+def mock_quopri(mocker):
+    return mocker.patch("embedchain.loaders.gmail.quopri.decodestring", return_value=b"your_test_decoded_string")
+
+
+@pytest.fixture
+def mock_beautifulsoup(mocker):
+    return mocker.patch("embedchain.loaders.gmail.BeautifulSoup", return_value=mocker.MagicMock())
+
+
+@pytest.fixture
+def gmail_loader(mock_download_loader, mock_quopri, mock_beautifulsoup):
+    return GMAILLoader()
+
+
+def test_load_data_file_not_found(gmail_loader, mocker):
+    with pytest.raises(FileNotFoundError):
+        with mocker.patch("os.path.isfile", return_value=False):
+            gmail_loader.load_data("your_query")
+
+
+def test_load_data(gmail_loader, mock_download_loader, mocker):
+    mock_gmail_reader_instance = mocker.MagicMock()
+    text = "your_test_email_text"
+    metadata = {
+        "id": "your_test_id",
+        "snippet": "your_test_snippet",
+    }
+    mock_gmail_reader_instance.load_data.return_value = [Document(text=text, extra_info=metadata)]
+    mock_download_loader.return_value = mock_gmail_reader_instance
+
+    with mocker.patch("os.path.isfile", return_value=True):
+        response_data = gmail_loader.load_data("your_query")
+
+    assert "doc_id" in response_data
+    assert "data" in response_data
+    assert isinstance(response_data["doc_id"], str)
+    assert isinstance(response_data["data"], list)


Code review comments:
diff_hunk: @@ -0,0 +1,35 @@
+---
+title: '📬 GMAIL'
path: docs/data-sources/gmail.mdx
text: @user1:
```suggestion
title: '📬 Gmail'
```

diff_hunk: @@ -0,0 +1,124 @@
+import hashlib
+import logging
+import os
+import quopri
+from textwrap import dedent
+
+from bs4 import BeautifulSoup
+
+try:
+    from llama_index import download_loader
+except ImportError:
+    raise ImportError("Notion requires extra dependencies. Install with `pip install embedchain[community]`") from None
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+def get_header(text: str, header: str) -> str:
+    start_string_position = text.find(header)
+    pos_start = text.find(":", start_string_position) + 1
+    pos_end = text.find("\n", pos_start)
+    header = text[pos_start:pos_end]
+    return header.strip()
+
+
+class GMAILLoader(BaseLoader):
+    def load_data(self, query):
+        """Load data from gmail."""
+        if not os.path.isfile("credentials.json"):
+            raise FileNotFoundError(
+                "You must download the valid credentials file from your google \
+                dev account. Refer this `https://cloud.google.com/docs/authentication/api-keys`"
+            )
+
+        GmailReader = download_loader("GmailReader")
+        loader = GmailReader(query=query, service=None, results_per_page=20)
+        documents = loader.load_data()
+        logging.info(f"GMAIL Loader: {len(documents)} mails found for query- {query}")
path: embedchain/loaders/gmail.py
text: @user1:
```suggestion
        logging.info(f"Gmail Loader: {len(documents)} mails found for query- {query}")
```

diff_hunk: @@ -168,6 +168,13 @@ dataloaders=[
     "pytube",
     "sentence-transformers",
     "unstructured",
+    "jq",
+    "requests",
+    "google-api-python-client",
+    "google-auth",
+    "google-auth-oauthlib",
+    "google-auth-httplib2",
+    "google-api-core",
 ]
path: pyproject.toml
text: @user1:
Can we please create a new `extra` called `gmail` and add all these loaders there as well? 

@author:
sure

diff_hunk: @@ -0,0 +1,124 @@
+import hashlib
+import logging
+import os
+import quopri
+from textwrap import dedent
+
+from bs4 import BeautifulSoup
+
+try:
+    from llama_index import download_loader
+except ImportError:
+    raise ImportError("Notion requires extra dependencies. Install with `pip install embedchain[community]`") from None
+
+from embedchain.loaders.base_loader import BaseLoader
+from embedchain.utils import clean_string
+
+
+def get_header(text: str, header: str) -> str:
+    start_string_position = text.find(header)
+    pos_start = text.find(":", start_string_position) + 1
+    pos_end = text.find("\n", pos_start)
+    header = text[pos_start:pos_end]
+    return header.strip()
+
+
+class GMAILLoader(BaseLoader):
path: embedchain/loaders/gmail.py
text: @user1:
```suggestion
class GmailLoader(BaseLoader):
```

diff_hunk: @@ -0,0 +1,35 @@
+---
+title: '📬 GMAIL'
+---
+
+To use GMAILLoader you must install the extra dependencies with `pip install --upgrade embedchain[dataloaders]`.
path: docs/data-sources/gmail.mdx
text: @user1:
Can we please do `GmailLoader` everywhere instead of `GMAILLoader` ? 

@author:
Cool

diff_hunk: @@ -176,3 +176,8 @@ notebooks/*.yaml
 .ipynb_checkpoints/
 
 !configs/*.yaml
+
+
+# Google API
+credentials.json
+token.json
path: .gitignore
text: @user1:
Hmm I think these shouldn't be here in the repo. User should keep these somewhere else. 

@author:
Yes, the requirement is it should be in the directory where they run the loader. So will update the doc to mention this pre-req and remove from .gitignore, thanks!

diff_hunk: @@ -0,0 +1,22 @@
+from typing import Optional
+
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from embedchain.chunkers.base_chunker import BaseChunker
+from embedchain.config.add_config import ChunkerConfig
+from embedchain.helper.json_serializable import register_deserializable
+
+
+@register_deserializable
+class GMAILChunker(BaseChunker):
path: embedchain/chunkers/gmail.py
text: @user1:
```suggestion
class GmailChunker(BaseChunker):
```