Repository: deepset-ai/haystack

Issue:
Docs: Update Crawler docs
See Discord thread: https://discord.com/channels/993534733298450452/1130098949739515974
Depth can be greater than 1, introduced in https://github.com/deepset-ai/haystack/pull/4623


Discussion:

Resolving patch:
diff --git a/haystack/nodes/connector/crawler.py b/haystack/nodes/connector/crawler.py
--- a/haystack/nodes/connector/crawler.py
+++ b/haystack/nodes/connector/crawler.py
@@ -58,9 +58,11 @@ def __init__(
         Init object with basic params for crawling (can be overwritten later).
 
         :param urls: List of http(s) address(es) (can also be supplied later when calling crawl())
-        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:
-            0: Only initial list of urls
-            1: Follow links found on the initial URLs (but no further)
+        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Can be any integer >= 0.  
+                                For example:  
+                                0: Only initial list of urls.  
+                                1: Follow links found on the initial URLs (but no further).  
+                                2: Additionally follow links found on the second-level URLs.
         :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.
             All URLs not matching at least one of the regular expressions will be dropped.
         :param id_hash_keys: Generate the document id from a custom list of strings that refer to the document's
@@ -153,9 +155,11 @@ def crawl(
         If no parameters are provided to this method, the instance attributes that were passed during __init__ will be used.
 
         :param urls: List of http addresses or single http address
-        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:
-                              0: Only initial list of urls
-                              1: Follow links found on the initial URLs (but no further)
+        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Can be any integer >= 0.  
+                                For example:  
+                                0: Only initial list of urls.  
+                                1: Follow links found on the initial URLs (but no further).  
+                                2: Additionally follow links found on the second-level URLs.
         :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.
                            All URLs not matching at least one of the regular expressions will be dropped.
         :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content
@@ -374,9 +378,11 @@ def run(  # type: ignore
 
         :param output_dir: Path for the directory to store files
         :param urls: List of http addresses or single http address
-        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Current options:
-                              0: Only initial list of urls
-                              1: Follow links found on the initial URLs (but no further)
+        :param crawler_depth: How many sublinks to follow from the initial list of URLs. Can be any integer >= 0.  
+                                For example:  
+                                0: Only initial list of urls.  
+                                1: Follow links found on the initial URLs (but no further).  
+                                2: Additionally follow links found on the second-level URLs.
         :param filter_urls: Optional list of regular expressions that the crawled URLs must comply with.
                            All URLs not matching at least one of the regular expressions will be dropped.
         :param overwrite_existing_files: Whether to overwrite existing files in output_dir with new content


Test patch: