Repository: deepset-ai/haystack

Issue:
increase crawler depth to crawl all child pages 
can we have a default crawler depth so that crawler can traverse all child pages until no child page left. similar to Depth First Traversal.  


Discussion:
Hello @krishna408 I understand that the current limitation of crawling depth to 1 is problematic. We will add this issue to the backlog but I think it is a good issue for community contributions. If anyone from the community wants to work on improving the crawler let us know here and we will be happy to support you.
Was just thinking of trying working on this as a first issue. Do you have any suggestion on where to start for this particular issue @julian-risch?
@Tsadoq that's great to hear! Yes, I think the only part of the code that definitely needs to change is the if-block and the for loop here: https://github.com/deepset-ai/haystack/blob/34b7db020935a0eb1cc9e984540ab735b88a5f22/haystack/nodes/connector/crawler.py#L246

Right now, we have the variable `urls`, which contains all the urls that need to be visited. I think, one way to change the code is to have three lists or sets with 1) `urls_to_visit`, 2) `urls_visited` and 3) `urls_on_next_depth_level` instead.
How I imagine it: When we start crawling, the user specified at least one url in `urls_to_visit`. Every link we find on this page needs to be added to `urls_on_next_depth_level`.  And we need to remove urls from `urls_to_visit` and add them to `urls_visited` while we go through the urls specified by the user in `urls_to_visit`. This is the first level of depth. Once `urls_to_visit` is empty, we need to check whether the maximum depth specified by the user is reached. If not, we start another iteration by switching the three lists/sets in the following way:
`urls_to_visit` is filled with urls from `urls_on_next_depth_level`. `urls_on_next_depth_level` is set to an empty list/set. `urls_visited` remains as is. We can use it to filter out pages from `urls_to_visit` that we already visited. Now we can repeat going through `urls_to_visit`, removing step by step one url from that list and adding it to `urls_visited`. And adding any newly found links to `urls_on_next_depth_level`.
Does that makes sense to you? It's basically breadth-first search. Changes to `_extract_sublinks_from_url` for example should not be necessary.
> Does that makes sense to you? It's basically breadth-first search. Changes to _extract_sublinks_from_url for example should not be necessary.

make quite a lot of sense to me, I'll work on it!
I was wondering if this has been worked on , would like to contribute
There is no open PR in Haystack addressing this issue and we didn't hear from @Tsadoq again. The only open PR about the crawler is this unrelated one: https://github.com/deepset-ai/haystack/pull/3838
@Ok3ks feel free to start working on it! 👍 I would encourage you to open a draft PR early on so that we can give feedback. Here are our guidelines for contributors: https://github.com/deepset-ai/haystack/blob/main/CONTRIBUTING.md
Would like to start contributing to Haystack starting with this issue 😊. I will submit my PR for this in about one week.
@benheckmann That sound great! Looking forward to it!

Resolving patch:
diff --git a/haystack/nodes/connector/crawler.py b/haystack/nodes/connector/crawler.py
--- a/haystack/nodes/connector/crawler.py
+++ b/haystack/nodes/connector/crawler.py
@@ -6,7 +6,7 @@
 import sys
 import time
 from pathlib import Path
-from typing import Any, Callable, Dict, List, Optional, Tuple, Union
+from typing import Any, Callable, Dict, List, Optional, Tuple, Union, Set
 from urllib.parse import urlparse
 
 try:
@@ -262,51 +262,18 @@ def crawl(
                 logger.info("Fetching from %s to `%s`", urls, output_dir)
 
         documents: List[Document] = []
-
-        # Start by crawling the initial list of urls
-        if filter_urls:
-            pattern = re.compile("|".join(filter_urls))
-            for url in urls:
-                if pattern.search(url):
-                    documents += self._crawl_urls(
-                        [url],
-                        extract_hidden_text=extract_hidden_text,
-                        loading_wait_time=loading_wait_time,
-                        id_hash_keys=id_hash_keys,
-                        output_dir=output_dir,
-                        overwrite_existing_files=overwrite_existing_files,
-                        file_path_meta_field_name=file_path_meta_field_name,
-                        crawler_naming_function=crawler_naming_function,
-                    )
-        else:
-            documents += self._crawl_urls(
-                urls,
-                extract_hidden_text=extract_hidden_text,
-                loading_wait_time=loading_wait_time,
-                id_hash_keys=id_hash_keys,
-                output_dir=output_dir,
-                overwrite_existing_files=overwrite_existing_files,
-                file_path_meta_field_name=file_path_meta_field_name,
-                crawler_naming_function=crawler_naming_function,
-            )
-
-        # follow one level of sublinks if requested
-        if crawler_depth == 1:
-            sub_links: Dict[str, List] = {}
-            for url_ in urls:
-                already_found_links: List = list(sum(list(sub_links.values()), []))
-                sub_links[url_] = list(
-                    self._extract_sublinks_from_url(
-                        base_url=url_,
-                        filter_urls=filter_urls,
-                        already_found_links=already_found_links,
-                        loading_wait_time=loading_wait_time,
+        uncrawled_urls = {base_url: {base_url} for base_url in urls}
+        crawled_urls = set()
+        for current_depth in range(crawler_depth + 1):
+            for base_url, uncrawled_urls_for_base in uncrawled_urls.items():
+                urls_to_crawl = list(
+                    filter(
+                        lambda u: (not filter_urls or re.search("|".join(filter_urls), u)) and u not in crawled_urls,
+                        uncrawled_urls_for_base,
                     )
                 )
-            for url, extracted_sublink in sub_links.items():
-                documents += self._crawl_urls(
-                    extracted_sublink,
-                    base_url=url,
+                crawled_documents = self._crawl_urls(
+                    urls_to_crawl,
                     extract_hidden_text=extract_hidden_text,
                     loading_wait_time=loading_wait_time,
                     id_hash_keys=id_hash_keys,
@@ -315,7 +282,19 @@ def crawl(
                     file_path_meta_field_name=file_path_meta_field_name,
                     crawler_naming_function=crawler_naming_function,
                 )
-
+                documents += crawled_documents
+                crawled_urls.update(urls_to_crawl)
+                if current_depth < crawler_depth:
+                    uncrawled_urls[base_url] = set()
+                    for url_ in urls_to_crawl:
+                        uncrawled_urls[base_url].update(
+                            self._extract_sublinks_from_url(
+                                base_url=url_,
+                                filter_urls=filter_urls,
+                                already_found_links=list(crawled_urls),
+                                loading_wait_time=loading_wait_time,
+                            )
+                        )
         return documents
 
     def _create_document(
@@ -527,7 +506,7 @@ def _extract_sublinks_from_url(
         filter_urls: Optional[List] = None,
         already_found_links: Optional[List] = None,
         loading_wait_time: Optional[int] = None,
-    ) -> set:
+    ) -> Set[str]:
         self.driver.get(base_url)
         if loading_wait_time is not None:
             time.sleep(loading_wait_time)


Test patch:
diff --git a/test/nodes/test_connector.py b/test/nodes/test_connector.py
--- a/test/nodes/test_connector.py
+++ b/test/nodes/test_connector.py
@@ -140,12 +140,6 @@ def test_crawler_filter_urls(test_url, tmp_path):
     assert len(documents) == 1
     assert content_match(crawler, test_url + "/index.html", documents[0].meta["file_path"])
 
-    # Note: filter_urls can exclude pages listed in `urls` as well
-    documents = crawler.crawl(urls=[test_url + "/index.html"], filter_urls=["page1"], crawler_depth=1)
-    assert len(documents) == 1
-    assert content_match(crawler, test_url + "/page1.html", documents[0].meta["file_path"])
-    assert not crawler.crawl(urls=[test_url + "/index.html"], filter_urls=["google.com"], crawler_depth=1)
-
 
 @pytest.mark.integration
 def test_crawler_extract_hidden_text(test_url, tmp_path):
@@ -238,3 +232,32 @@ def test_crawler_custom_meta_file_path_name(test_url, tmp_path):
         urls=[test_url + "/index.html"], crawler_depth=0, output_dir=tmp_path, file_path_meta_field_name="custom"
     )
     assert documents[0].meta.get("custom", None) is not None
+
+
+@pytest.mark.integration
+def test_crawler_depth_2_single_url(test_url, tmp_path):
+    crawler = Crawler(output_dir=tmp_path, file_path_meta_field_name="file_path")
+    documents = crawler.crawl(urls=[test_url + "/index.html"], crawler_depth=2)
+    assert len(documents) == 6
+    paths = [doc.meta["file_path"] for doc in documents]
+    assert content_in_results(crawler, test_url + "/index.html", paths)
+    assert content_in_results(crawler, test_url + "/page1.html", paths)
+    assert content_in_results(crawler, test_url + "/page2.html", paths)
+    assert content_in_results(crawler, test_url + "/page1_subpage1.html", paths)
+    assert content_in_results(crawler, test_url + "/page1_subpage2.html", paths)
+    assert content_in_results(crawler, test_url + "/page2_subpage1.html", paths)
+
+
+@pytest.mark.integration
+def test_crawler_depth_2_multiple_urls(test_url, tmp_path):
+    crawler = Crawler(output_dir=tmp_path, file_path_meta_field_name="file_path")
+    _urls = [test_url + "/index.html", test_url + "/page1.html"]
+    documents = crawler.crawl(urls=_urls, crawler_depth=2)
+    assert len(documents) == 6
+    paths = [doc.meta["file_path"] for doc in documents]
+    assert content_in_results(crawler, test_url + "/index.html", paths)
+    assert content_in_results(crawler, test_url + "/page1.html", paths)
+    assert content_in_results(crawler, test_url + "/page2.html", paths)
+    assert content_in_results(crawler, test_url + "/page1_subpage1.html", paths)
+    assert content_in_results(crawler, test_url + "/page1_subpage2.html", paths)
+    assert content_in_results(crawler, test_url + "/page2_subpage1.html", paths)
diff --git a/test/samples/crawler/page1.html b/test/samples/crawler/page1.html
--- a/test/samples/crawler/page1.html
+++ b/test/samples/crawler/page1.html
@@ -7,5 +7,7 @@
     <p>page 1 content</p>
     <a href="index.html">link to home</a>
     <a href="page2.html">link to page 2</a>
+    <a href="page1_subpage1.html">link to subpage 1</a>
+    <a href="page1_subpage2.html">link to subpage 2</a>
 </body>
 </html>
diff --git a/test/samples/crawler/page1_subpage1.html b/test/samples/crawler/page1_subpage1.html
new file mode 100644
--- /dev/null
+++ b/test/samples/crawler/page1_subpage1.html
@@ -0,0 +1,9 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Test Subpage 1 of Page 1 for Crawler</title>
+</head>
+<body>
+    <p>subpage 1 of page 1 content</p>
+</body>
+</html>
diff --git a/test/samples/crawler/page1_subpage2.html b/test/samples/crawler/page1_subpage2.html
new file mode 100644
--- /dev/null
+++ b/test/samples/crawler/page1_subpage2.html
@@ -0,0 +1,9 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Test Subpage 2 of Page 1 for Crawler</title>
+</head>
+<body>
+    <p>subpage 2 of page 1 content</p>
+</body>
+</html>
diff --git a/test/samples/crawler/page2.html b/test/samples/crawler/page2.html
--- a/test/samples/crawler/page2.html
+++ b/test/samples/crawler/page2.html
@@ -7,5 +7,6 @@
     <p>page 2 content</p>
     <a href="index.html">link to home</a>
     <a href="page1.html">link to page 1</a>
+    <a href="page2_subpage1.html">link to subpage</a>
 </body>
 </html>
diff --git a/test/samples/crawler/page2_subpage1.html b/test/samples/crawler/page2_subpage1.html
new file mode 100644
--- /dev/null
+++ b/test/samples/crawler/page2_subpage1.html
@@ -0,0 +1,9 @@
+<!DOCTYPE html>
+<html>
+<head>
+    <title>Test Subpage 1 of Page 2 for Crawler</title>
+</head>
+<body>
+    <p>subpage 1 of page 2 content</p>
+</body>
+</html>