Repository: deepset-ai/haystack

Issue:
Crawler test on Windows sometimes fails
**Describe the bug**
I've found that the Crawler test `test_crawler` on the integration tests for Windows can sometimes fail. 

**Error message**
```bash
tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_crawler0')

    @pytest.mark.integration
    def test_crawler(tmp_path):
        tmp_dir = tmp_path
        url = ["https://haystack.deepset.ai/"]
    
>       crawler = Crawler(output_dir=tmp_dir)

test\nodes\test_connector.py:67: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
haystack\nodes\base.py:48: in wrapper_exportable_to_yaml
    init_func(self, *args, **kwargs)
haystack\nodes\connector\crawler.py:164: in __init__
    self.driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\selenium\webdriver\chrome\webdriver.py:89: in __init__
    keep_alive,
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\selenium\webdriver\chromium\webdriver.py:112: in __init__
    options=options,
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\selenium\webdriver\remote\webdriver.py:286: in __init__
    self.start_session(capabilities, browser_profile)
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\selenium\webdriver\remote\webdriver.py:378: in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\selenium\webdriver\remote\webdriver.py:440: in execute
    self.error_handler.check_response(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x0000016C7D25B588>
response = {'status': 500, 'value': '{"value":{"error":"unknown error","message":"unknown error: DevToolsActivePort file doesn\'t...Thunk [0x77A86A39+25]\\n\\tRtlGetFullPathName_UEx [0x77DC7D1D+173]\\n\\tRtlGetFullPathName_UEx [0x77DC7CEB+123]\\n"}}'}
```

**Expected behavior**
For the tests to pass. 

**Additional context**
- This was found during the CI of the PR 
- The test in question was recently added to windows CI in this [PR](https://github.com/deepset-ai/haystack/pull/3830/files).

**To Reproduce**
Not entirely sure when this error occurs in the Windows CI. Some additional info on this error can be found at this Stack Overlfow [post](https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t). 

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://docs.haystack.deepset.ai/docs/faq)?

**System:**
 - OS: Windows CI
 - GPU/CPU: CPU
 - Haystack version (commit or version number): 
 - DocumentStore: None
 - Reader: None
 - Retriever: None



Discussion:

Resolving patch:
diff --git a/haystack/nodes/connector/crawler.py b/haystack/nodes/connector/crawler.py
--- a/haystack/nodes/connector/crawler.py
+++ b/haystack/nodes/connector/crawler.py
@@ -95,27 +95,28 @@ def __init__(
                     and spawn a single process.
                  2) ["--no-sandbox"]
                     This option disables the sandbox, which is required for running Chrome as root.
-            See [Chrome Web Driver Options](https://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.chrome.options) for more details.
+                 3) ["--remote-debugging-port=9222"]
+                    This option enables remote debug over HTTP.
+            See [Chromium Command Line Switches](https://peter.sh/experiments/chromium-command-line-switches/) for more details on the available options.
+            If your crawler fails, rasing a `selenium.WebDriverException`, this [Stack Overflow thread](https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t) can be helpful. Contains useful suggestions for webdriver_options.
         """
         super().__init__()
 
         IN_COLAB = "google.colab" in sys.modules
         IN_AZUREML = os.environ.get("AZUREML_ENVIRONMENT_IMAGE", None) == "True"
-        IS_ROOT = sys.platform not in ["win32", "cygwin"] and os.geteuid() == 0  # type: ignore   # This is a mypy issue of sorts, that fails on Windows.
+        IN_WINDOWS = sys.platform in ["win32", "cygwin"]
+        IS_ROOT = not IN_WINDOWS and os.geteuid() == 0  # type: ignore   # This is a mypy issue of sorts, that fails on Windows.
 
         if webdriver_options is None:
             webdriver_options = ["--headless", "--disable-gpu", "--disable-dev-shm-usage", "--single-process"]
-        elif "--headless" not in webdriver_options:
-            webdriver_options.append("--headless")
-
-        if IS_ROOT and "--no-sandbox" not in webdriver_options:
-            webdriver_options.append("--no-sandbox")
-
-        if (IN_COLAB or IN_AZUREML) and "--disable-dev-shm-usage" not in webdriver_options:
+        webdriver_options.append("--headless")
+        if IS_ROOT or IN_WINDOWS:
+            webdriver_options.extend(["--no-sandbox", "--remote-debugging-port=9222"])
+        if IN_COLAB or IN_AZUREML:
             webdriver_options.append("--disable-dev-shm-usage")
 
         options = Options()
-        for option in webdriver_options:
+        for option in set(webdriver_options):
             options.add_argument(option)
 
         if IN_COLAB:


Test patch: