Repository: kaizendorks/pymongo_inmemory

Issue:
pymongo_inmemory does not work in a docker container when /tmp directory is mounted. No bypass available.
**Describe the bug**
I have a complicated CLI docker container that I share with my developers. In it, I have created a kubernetes plugin, `kubectl query` which allows us to load internal data into an in-memory and search it. It works great and as expected when running on MacOS.

When packaged and launched our docker container, we mount the `/tmp` directory as a method of porting data inbetween the Mac and the Ubuntu container. This allows us to easily transfer our scratch data between the OS in Ubuntu and Mac. This is often used for other kubectl plugins that we have written.

The fact that we are mounting the `/tmp` directory is not playing well with `pymongo_inmemory`. I have discovered that if we opt not to mount the `/tmp` directory, then `pymongo_inmemory` correctly starts. Only when we mount the `/tmp` directory, does `pymongo_inmemory` fail to connect. *This is reproducible with the code below*.

I have attempted furthermore to bypass this issue by setting the `TMPDIR`, `TEMP`, and `TMP` directories, so that the `tempfile` package ignores the volume mounting. However, I am still exhibiting this problem.

**To Reproduce**
Steps to reproduce the behavior:

1. Create a Dockerfile:
```
FROM ubuntu:20.04
RUN apt-get update && apt-get -y install \
  bash \
  bash-completion \
  bash-doc \
  python3 \
  python3-dev \
  python3-pip
RUN pip3 install --upgrade pymongo_inmemory==0.3.0
RUN echo '\n\
from pymongo_inmemory import MongoClient \n\
client = MongoClient() \n\
db = client["testdb"] \n\
collection = db["test-collection"] \n\
# etc., etc. \n\
client.close() \n'\
> /query.py
CMD /bin/bash
```
2. Build Dockerfile
```
docker build ./ -t temp --platform=linux/x86_64
```
3. Run Docker file, everything will be OK:
```
docker run --platform=linux/x86_64 -it temp python3 query.py
```
4. Run Docker file, override mount, this will fail:
```
docker run -v /tmp:/tmp --platform=linux/x86_64 -it temp python3 query.py
```
5. Attempt patching with different tempdir, Run Docker file, override mount, this will fail:
```
docker run -e TMPDIR=/pytemp -v /tmp:/tmp  --platform=linux/x86_64 -it temp python3 query.py
```
*Full Error*
```
$ docker run -e TMPDIR=/pytemp -v /tmp:/tmp  --platform=linux/x86_64 -it temp python3 query.py
Starting from MongoDB 4.0.23 there isn't a generic Linux version of MongoDB
Traceback (most recent call last):
  File "query.py", line 3, in <module>
    client = MongoClient() 
  File "/usr/local/lib/python3.8/dist-packages/pymongo_inmemory/_pim.py", line 11, in __init__
    self._mongod.start()
  File "/usr/local/lib/python3.8/dist-packages/pymongo_inmemory/mongod.py", line 130, in start
    while not self.is_healthy:
  File "/usr/local/lib/python3.8/dist-packages/pymongo_inmemory/mongod.py", line 164, in is_healthy
    status = db.command("serverStatus")
  File "/usr/local/lib/python3.8/dist-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/database.py", line 892, in command
    with self.__client._conn_for_reads(read_preference, session) as (
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1340, in _conn_for_reads
    server = self._select_server(read_preference, session)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/mongo_client.py", line 1297, in _select_server
    server = topology.select_server(server_selector)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 312, in select_server
    server = self._select_server(selector, server_selection_timeout, address)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 296, in _select_server
    servers = self.select_servers(selector, server_selection_timeout, address)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 247, in select_servers
    server_descriptions = self._select_servers_loop(selector, server_timeout, address)
  File "/usr/local/lib/python3.8/dist-packages/pymongo/topology.py", line 269, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 650b42c35fed3c72b372d834, topology_type: Unknown, servers: [<ServerDescription ('127.0.0.1', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('127.0.0.1:27017: [Errno 111] Connection refused')>]>
```

**Expected behavior**
I was expecting successful startup of Mongo DB:
```
$ docker run --platform=linux/x86_64 -it temp python3 query.py
Starting from MongoDB 4.0.23 there isn't a generic Linux version of MongoDB
```

In the event that MongoDB can't share the `/tmp` directory with MacOS, I was expecting `TMPDIR` to get the system working. *NOTE:* `pymongo_inmemory` *is* respecting the `TMPDIR` environment variable, it just isn't starting Mongodb, and I'm not sure why.

**Logs**
See above error.

**Screenshots**
N/A

**Context:**
 - OS: Apple M1 Pro, Ventura 13.4.1 w/ Docker Container ubuntu:20.04
 - Version of pymongo_inmemory: 0.3.0
 - Version of mongo you are downloading: 4.0.23
 - Version of Docker:  4.20.1 (110738)
 - Any other additional information about context
 
Workarounds instead of direct fixes are acceptable to me.

Discussion:

Patch:
diff --git a/README.md b/README.md
index 1e6b132..18ef7ba 100644
--- a/README.md
+++ b/README.md
@@ -7,6 +7,7 @@ A mongo mocking library with an ephemeral MongoDB running in memory.
 ## What's new?
 
 ### v0.4.0-pre
+
 - Tooling enhancements. [[PR #90](https://github.com/kaizendorks/pymongo_inmemory/pull/90)]
 
 ### v0.3.1
@@ -60,18 +61,19 @@ with MongoClient() as client:
 
 ## Configuration
 
-| Config param       | Description                                                                                                | Optional? | Default                                                                                                                    |
-| ------------------ | ---------------------------------------------------------------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------- |
-| `mongo_version`    | Which MongoD version to download and use.                                                                  | Yes       | Latest for the OS                                                                                                          |
-| `mongod_port`      | Override port preference.                                                                                  | Yes       | Automatically picked between `27017` and `28000` after testing availability                                                |
-| `operating_system` | This makes sense for Linux setting, where there are several flavours                                       | Yes       | Automatically determined (Generic for Linux)\*                                                                             |
-| `os_version`       | If an operating system has several versions use this parameter to select one                               | Yes       | Latest version of the OS will be selected from the list                                                                    |
-| `download_url`     | If set, it won't attempt to determine which MongoDB to download. However there won't be a fallback either. | Yes       | Automatically determined from given parameters and using [internal URL bank](pymongo_inmemory/downloader/_patterns.py)\*\* |
-| `ignore_cache`     | Even if there is a downloaded version in the cache, download it again.                                     | Yes       | False                                                                                                                      |
-| `use_local_mongod` | If set, it will try to use a local mongod instance instead of downloading one.                             | Yes       | False                                                                                                                      |
-| `download_folder`  | Override the default download location.                                                                    | Yes       | pymongo_inmemory/.cache/download                                                                                           |
-| `extract_folder`   | Override the default extraction location.                                                                  | Yes       | pymongo_inmemory/.cache/extract                                                                                            |
-|                    |                                                                                                            |           |
+|         | Config parameter     | Description Default                                                                                        |
+| ------- | -------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
+|         | `mongo_version`      | Which MongoD version to download and use.                                                                  | Latest for the OS                                                                                                          |
+|         | `mongod_port`        | Override port preference.                                                                                  | Automatically picked between `27017` and `28000` after testing availability                                                |
+|         | `operating_system`   | This makes sense for Linux setting, where there are several flavours                                       | Automatically determined (Generic for Linux)\*                                                                             |
+|         | `os_version`         | If an operating system has several versions use this parameter to select one                               | Latest version of the OS will be selected from the list                                                                    |
+|         | `download_url`       | If set, it won't attempt to determine which MongoDB to download. However there won't be a fallback either. | Automatically determined from given parameters and using [internal URL bank](pymongo_inmemory/downloader/_patterns.py)\*\* |
+|         | `ignore_cache`       | Even if there is a downloaded version in the cache, download it again.                                     | False                                                                                                                      |
+|         | `use_local_mongod`   | If set, it will try to use a local mongod instance instead of downloading one.                             | False                                                                                                                      |
+|         | `download_folder`    | Override the default download location.                                                                    | pymongo_inmemory/.cache/download                                                                                           |
+|         | `extract_folder`     | Override the default extraction location.                                                                  | pymongo_inmemory/.cache/extract                                                                                            |
+| **NEW** | `mongod_data_folder` | Provide a data folder to be used by MongoD.                                                                | A `TemporaryDirectory` will be used                                                                                        |
+|         |                      |                                                                                                            |
 
 - \***_Note 1:_** Generic Linux version offering for MongoDB ends with version **4.0.23**. If the operating system is just `linux` and if selected MongoDB version is higher, it will default to `4.0.23`.
 - **\***Note 2:\*\*\* URL bank is filled with URLs collected from [release list](https://www.mongodb.com/download-center/community/releases) and [archived released list](https://www.mongodb.com/download-center/community/releases/archive), so if a version is not in the bank you can use the same list to provide an official download link.
diff --git a/pymongo_inmemory/context.py b/pymongo_inmemory/context.py
index 00f6e57..e52083c 100644
--- a/pymongo_inmemory/context.py
+++ b/pymongo_inmemory/context.py
@@ -103,6 +103,7 @@ class Context:
     ) -> None:
         self.mongo_version = conf("mongo_version", version)
         self.mongod_port = conf("mongod_port", None, coerce_with=int)
+        self.mongod_data_folder = conf("mongod_data_folder", None)
 
         self.operating_system = self._build_operating_system_info(os_name)
         self.os_version = conf("os_version", os_ver)
@@ -129,6 +130,7 @@ class Context:
         return (
             f"Mongo Version {self.mongo_version}\n"
             f"MongoD Port {self.mongod_port}\n"
+            f"MongoD Data Folder {self.mongod_data_folder}\n"
             f"OS Name {self.operating_system}\n"
             f"OS Version {self.os_version}\n"
             f"Download URL {self.download_url}\n"
diff --git a/pymongo_inmemory/mongod.py b/pymongo_inmemory/mongod.py
index c09ce6f..6b0fc58 100644
--- a/pymongo_inmemory/mongod.py
+++ b/pymongo_inmemory/mongod.py
@@ -87,7 +87,10 @@ class Mongod:
         self._connection_string = None
 
         self.config = MongodConfig(self._pim_context)
-        self.data_folder = TemporaryDirectory(prefix="pymongoim")
+
+        self._temp_data_folder = TemporaryDirectory(prefix="pymongoim")
+        self._using_tmp_folder = self._pim_context.mongod_data_folder is None
+
         self._client = pymongo.MongoClient(self.connection_string)
 
     def __enter__(self):
@@ -98,22 +101,14 @@ class Mongod:
         self.stop()
 
     def start(self):
-        while self.is_locked:
-            logger.warning(
-                (
-                    "Lock file found, possibly another mock server is running. "
-                    "Changing the data folder."
-                )
-            )
-            self.data_folder = TemporaryDirectory(prefix="pymongoim")
-
-        self.log_path = os.path.join(self.data_folder.name, "mongod.log")
+        self._check_lock()
+        self.log_path = os.path.join(self.data_folder, "mongod.log")
 
         logger.info("Starting mongod with {cs}...".format(cs=self.connection_string))
         boot_command = [
             os.path.join(self._bin_folder, "mongod"),
             "--dbpath",
-            self.data_folder.name,
+            self.data_folder,
             "--logpath",
             self.log_path,
             "--port",
@@ -137,7 +132,14 @@ class Mongod:
         while self._proc.poll() is None:
             logger.debug("Waiting for MongoD shutdown.")
             time.sleep(1)
-        self.data_folder.cleanup()
+        self._clean_up()
+
+    @property
+    def data_folder(self):
+        if self._using_tmp_folder:
+            return self._temp_data_folder.name
+        else:
+            return self._pim_context.mongod_data_folder
 
     @property
     def connection_string(self):
@@ -155,7 +157,7 @@ class Mongod:
 
     @property
     def is_locked(self):
-        return os.path.exists(os.path.join(self.data_folder.name, "mongod.lock"))
+        return os.path.exists(os.path.join(self.data_folder, "mongod.lock"))
 
     @property
     def is_healthy(self):
@@ -199,6 +201,27 @@ class Mongod:
         with open(self.log_path, "r") as logfile:
             return logfile.readlines()
 
+    def _clean_up(self):
+        if self._using_tmp_folder:
+            self._temp_data_folder.cleanup()
+
+    def _check_lock(self):
+        while self.is_locked:
+            if self._using_tmp_folder:
+                raise RuntimeError(
+                    (
+                        "There is a lock file in the provided data folder. "
+                        "Make sure that no other MongoDB is running."
+                    )
+                )
+            logger.warning(
+                (
+                    "Lock file found, possibly another mock server is running. "
+                    "Changing the data folder."
+                )
+            )
+            self._temp_data_folder = TemporaryDirectory(prefix="pymongoim")
+
 
 if __name__ == "__main__":
     # This part is used for integrity tests too.


Test patch:
diff --git a/tests/unit/test_mongod.py b/tests/unit/test_mongod.py
new file mode 100644
index 0000000..986c7be
--- /dev/null
+++ b/tests/unit/test_mongod.py
@@ -0,0 +1,37 @@
+import subprocess
+
+from pymongo_inmemory.mongod import Mongod
+import pymongo_inmemory.downloader as downloader
+from pymongo_inmemory.context import Context
+
+
+class Popen:
+    def __init__(self, cmd):
+        self.cmd = cmd
+        self.terminated = False
+
+    def terminate(self):
+        self.terminated = True
+
+    def poll(self):
+        return True
+
+
+def returns_true():
+    return True
+
+
+def download():
+    return ""
+
+
+def test_mongod(monkeypatch):
+    monkeypatch.setattr(subprocess, "Popen", Popen)
+    monkeypatch.setattr(Mongod, "is_healthy", returns_true)
+    monkeypatch.setattr(downloader, "download", download)
+
+    context = Context()
+    context.mongod_data_folder = "TEST"
+
+    with Mongod(context) as md:
+        assert md.data_folder == "TEST"