Repository: deepset-ai/haystack

Issue:
Crawler is crashing in some container environments
**Describe the bug**
Crawler is crashing when running on some containers, e.g., Azure ML and OpenShift.

**Error message**
In a WebDriver exception, Chrome driver detect Chrome stopped responding, and say it probably crashed.

**Expected behavior**
Crawler should run without issues in containers.

**Investigation**
After some investigation, this points to be a similar error that occurred in Google Colab environment. Without proper flags, Chrome will try to use /dev/shm, and in most container images there is no permission for that (except if you set).
Going to [Selenium repository](https://github.com/SeleniumHQ/docker-selenium) where they store their Dockerfiles, show us their “recommended” setup for a container image. Some important files:

- [Base image](https://github.com/SeleniumHQ/docker-selenium/blob/5c0f740c4e75388ee0074dad640ead5ba7437fc9/NodeBase/Dockerfile)
- [Chrome image](https://github.com/SeleniumHQ/docker-selenium/blob/5c0f740c4e75388ee0074dad640ead5ba7437fc9/NodeChrome/Dockerfile)

The base image is the most important regarding crashes in general. Disabling shm, like on Colab code, make it unnecessary to set up permissions on shm for some environments.
Furthermore, I must make a note that sometimes Chrome crashes if you don't disable audio and pulseaudio is not installed (this has been tested on AWS Lambda container). So maybe haystack Dockerfile should include it, or Crawler should disable audio.
When using nvidia CUDA images, disabling Chrome GPU usage also decreased random crashes.
As a side note, using Chrome option to spawn a single process saves some memory.


I've made this [test branch](https://github.com/danielbichuetti/haystack/tree/test_azureml_crawler), running it on Azure ML, Openshift, EKS and AKS increased stability, without Crawler crashes. Obs: this is just a fast test branch, not the one supposed for a PR (if).

Perhaps the AzureML settings should be made default, since haystack doesn't allow user to configure Selenium WebDriver options. Setting the safest options would be, possibly, the recommended.

**To Reproduce**
Create a Dockerfile with Haystack (Crawler enabled), use it to generate an environment in Azure ML. Try to run Crawler using Jobs.

**FAQ Check**
- [x] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: Linux 
 - GPU/CPU: NVIDIA V100
 - Haystack version (commit or version number): 1.6.1rc



Discussion:
> Perhaps the AzureML settings should be made default, since haystack doesn't allow user to configure Selenium WebDriver options. Setting the safest options would be, possibly, the recommended.

Perhaps another solution would be to add `options` to the initialization of the `Crawler` so the user could configure Selenium WebDriver options. I agree we should find a reasonable set of default options, but it is unlikely we can find the "safest" options for all scenarios. 

Also what happens to Selenium when we disable `/dev/shm`? 
> Without proper flags, Chrome will try to use /dev/shm, and in most container images there is no permission for that (except if you set).

Do you mean users would need to run `chmod +x /dev/shm` before using the Crawler? Or is there also an issue of permissions where that `chmod` command could be rejected? We could consider adding a `try-except` to the `Crawler` to run that `chmod` command and if it fails pass the option `--disable-dev-shm-usage`.
There is a pretty good discussion on the issues the `chrome` webdriver has in docker environments in general https://github.com/puppeteer/puppeteer/issues/1834. It seems like there are some cons to having `--disable-dev-shm-usage` on by default, making me think allowing users to choose what options to pass is the way to go. And to add documentation warning about the common issues that one might face when running the Crawler in a docker environment.  
> Also what happens to Selenium when we disable /dev/shm?

Selenium will not change any behavior, the behavior will be changed only on Chrome (`--disable-dev-shm-usage`)

> Do you mean users would need to run chmod +x /dev/shm before using the Crawler? Or is there also an issue of permissions where that chmod command could be rejected?

Chrome will try to write to **/dev/shm** if no data and temp directory is provided. Without proper permissions, it will crash. Furthermore, this directory need to have proper size available ( **RAM** ).

At this [link](https://bugs.chromium.org/p/chromium/issues/detail?id=736452) there is one discussion about similar issues:

> Or change to /tmp or another directory instead of /dev/shm even when /dev/shm is available, but too small (e.g. Docker).

> It seems like there are some cons to having --disable-dev-shm-usage on by default, making me think allowing users to choose what options to pass is the way to go.

When you disable it, you are basically telling Chrome: write temporary data (e.g., profile, cache) to disk (/tmp, or a directory passed with parameter), and not to shared RAM. On most container environments, you can't write to shared RAM by default.

Maybe since haystack already is a heavy RAM framework if doing NLP, it would be better to **disable Crawler from using RAM and GPU**. So other **NLP tasks on the same machine won't be affected**. Chrome will use the disk.

I think the cons of **disabling /dev/shm** writing apply for testings scenarios (see quote below) like Selenium Grid and other where there are many processes and tabs (a dedicated testing environment), so **disk I/O may increase**. On **haystack** scenario, **using RAM or GPU**, on my perspective, is indeed **not the preferred option**.

> Hey Aslushnikov, I'll see when I have time to run a clean benchmark.
> 
> My setup consists of puppeteer pods running on Kubernetes. These pods autoscale to an average of 10 concurrent puppeteer requests. With an absolute max of 20, and everything higher is queued, but this only happens on very high spikes while the autoscaler needs to scale. Each puppeteer request consists of an incoming HTTP request, that starts a new chromium browser, executes the script and returns either a rendered html or a screenshot.
> 
> If these pods are allocated enough CPU, the next bottleneck will be the disk IO, especially in a docker container. Running using --disable-dev-shm-usage will only impact this bottleneck further. To be honest to mitigate this bottleneck I even went further and moved the chromium profiles and chromium executable to /dev/shm completely. Since there we're multiple browsers opening/closing every second, causing repetitive disk reads and writes.
> 
> This approach allowed me to scale for peaks up to 400 concurrent sessions, and if needed probably a lot higher.
> 
> Now while --disable-dev-shm-usage is a perfectly valid choice when you're just running small tasks, and performance doesn't matter that much. I ultimately think this is a choice that should be left to the developer itself. Especially since not everyone is running in docker to begin with. Otherwise we might start adding flags like `--disable-gpu` or `--no-sandbox` by default as well, since these also cause issues in docker.
> 
> My proposal would be to leave these flags out of any default configuration, but add a clear FAQ with docker caveats instead.

About the GPU option, you may see puppeteer devs comment in same [reference issue](https://github.com/puppeteer/puppeteer/issues/1834):

> In fact, we do [still add --disable-gpu](https://github.com/GoogleChrome/puppeteer/blob/18f2ecdffdfc70e891750b570bfe8bea5b5ca8c2/lib/Launcher.js#L91) because enough users were having problems across platforms.

Yet at the same referenced issue, many other issues referencing that issue are shown, and most projects have made the option to disable shared memory writing. But, also, there are problems when writing to disk on heavy testing scenarios. What makes me think about the default option (_"most"_ safe), and the possibility to set user-defined options. :thinking: 

What about allowing the user to send a list of parameters to be sent to Chrome, and if none sent, use a predefined set of options, which would be the safest for a scenario where we can't:
- control container image used
- control disk permissions applied (maybe too intrusive?)

And we, definitely, know that someone, who is using haystack, will be doing NLP, so the chances of **RAM/GPU being in trouble** are larger.

And a last point: **GPU instances are expensive in the cloud**. No need to allow a simple Crawler utility to use it.


Hey @sjrl and @danielbichuetti - As far as I can tell this Issue is related to PR #2921 which has been closed. I will close this issue since the PR was merged. Let me know if that's wrong and I will re-open
@TuanaCelik Hi

This issue is not related to #2921. I found it after that, when doing tests on Azure ML notebooks. Where it won't work by default. Furthermore, it won't work on other docker container environment due to not disabling shm access.

This issue is related to how the Chrome driver is set up on the Crawler init (the options being sent to Chrome process command-line)
@danielbichuetti  - Got it thanks for the context, I've re-opened it now :)
> What about allowing the user to send a list of parameters to be sent to Chrome, and if none sent, use a predefined set of options, which would be the safest for a scenario where we can't:

@danielbichuetti I agree this sounds like the best idea. Would you be willing to go ahead and open a PR for this? And for the predefined set of options, I think making it clear in the documentation which ones are being used would be very helpful. 
@sjrl Sure! I will open the PR. Thank you!

Resolving patch:
diff --git a/haystack/nodes/connector/crawler.py b/haystack/nodes/connector/crawler.py
--- a/haystack/nodes/connector/crawler.py
+++ b/haystack/nodes/connector/crawler.py
@@ -1,5 +1,6 @@
 from typing import Callable, List, Optional, Dict, Tuple, Union, Any
 
+import os
 import re
 import sys
 import json
@@ -12,6 +13,7 @@
 try:
     from webdriver_manager.chrome import ChromeDriverManager
     from selenium.webdriver.chrome.service import Service
+    from selenium.webdriver.chrome.options import Options
     from selenium.webdriver.common.by import By
     from selenium.common.exceptions import StaleElementReferenceException, WebDriverException
     from selenium import webdriver
@@ -56,6 +58,7 @@ def __init__(
         extract_hidden_text=True,
         loading_wait_time: Optional[int] = None,
         crawler_naming_function: Optional[Callable[[str, str], str]] = None,
+        webdriver_options: Optional[List[str]] = None,
     ):
         """
         Init object with basic params for crawling (can be overwritten later).
@@ -83,17 +86,40 @@ def __init__(
                     This example will generate a file name from the url by replacing all characters that are not allowed in file names with underscores.
                  2) crawler_naming_function=lambda url, page_content: hashlib.md5(f"{url}{page_content}".encode("utf-8")).hexdigest()
                     This example will generate a file name from the url and the page content by using the MD5 hash of the concatenation of the url and the page content.
+        :param webdriver_options: A list of options to send to Selenium webdriver. If none is provided,
+            Crawler uses, as a default option, a reasonable selection for operating locally, on restricted docker containers,
+            and avoids using GPU.
+            Crawler always appends the following option: "--headless"
+            For example: 1) ["--disable-gpu", "--no-sandbox", "--disable-dev-shm-usage", "--single-process"]
+                    These are the default options which disable GPU, disable shared memory usage
+                    and spawn a single process.
+                 2) ["--no-sandbox"]
+                    This option disables the sandbox, which is required for running Chrome as root.
+            See [Chrome Web Driver Options](https://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.chrome.options) for more details.
         """
         super().__init__()
 
         IN_COLAB = "google.colab" in sys.modules
+        IN_AZUREML = True if os.environ.get("AZUREML_ENVIRONMENT_IMAGE", None) == "True" else False
+        IS_ROOT = True if os.geteuid() == 0 else False
+
+        if webdriver_options is None:
+            webdriver_options = ["--headless", "--disable-gpu", "--disable-dev-shm-usage", "--single-process"]
+        elif "--headless" not in webdriver_options:
+            webdriver_options.append("--headless")
+
+        if IS_ROOT and "--no-sandbox" not in webdriver_options:
+            webdriver_options.append("--no-sandbox")
+
+        if (IN_COLAB or IN_AZUREML) and "--disable-dev-shm-usage" not in webdriver_options:
+            webdriver_options.append("--disable-dev-shm-usage")
+
+        options = Options()
+        for option in webdriver_options:
+            options.add_argument(option)
 
-        options = webdriver.chrome.options.Options()
-        options.add_argument("--headless")
         if IN_COLAB:
             try:
-                options.add_argument("--no-sandbox")
-                options.add_argument("--disable-dev-shm-usage")
                 self.driver = webdriver.Chrome(service=Service("chromedriver"), options=options)
             except WebDriverException as exc:
                 raise NodeError(


Test patch: