Repository: devanshshukla99/pytest-remote-response

Issue:
Fix database
Probably better to move to a thread safe dB, just to be safe.

Work towards reducing the footprint.


Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla) Report
> Merging [#28](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla) (14cba5c) into [main](https://codecov.io/gh/devanshshukla99/pytest-remote-response/commit/c724ec73a4bddca226c22e2aae33bed593200b99?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla) (c724ec7) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

```diff
@@           Coverage Diff           @@
##             main      #28   +/-   ##
=======================================
  Coverage   99.53%   99.53%           
=======================================
  Files          14       14           
  Lines         214      215    +1     
=======================================
+ Hits          213      214    +1     
  Misses          1        1           
```


| [Impacted Files](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla) | Coverage Δ | |
|---|---|---|
| [examples/response\_decorator\_urllib3.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-ZXhhbXBsZXMvcmVzcG9uc2VfZGVjb3JhdG9yX3VybGxpYjMucHk=) | `100.00% <ø> (ø)` | |
| [tests/test\_aiohttp.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-dGVzdHMvdGVzdF9haW9odHRwLnB5) | `100.00% <ø> (ø)` | |
| [tests/test\_examples.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-dGVzdHMvdGVzdF9leGFtcGxlcy5weQ==) | `100.00% <ø> (ø)` | |
| [tests/test\_requests.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-dGVzdHMvdGVzdF9yZXF1ZXN0cy5weQ==) | `100.00% <ø> (ø)` | |
| [tests/test\_urllib.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-dGVzdHMvdGVzdF91cmxsaWIucHk=) | `100.00% <ø> (ø)` | |
| [tests/test\_urllib3.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-dGVzdHMvdGVzdF91cmxsaWIzLnB5) | `100.00% <ø> (ø)` | |
| [examples/insert\_get\_database.py](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla#diff-ZXhhbXBsZXMvaW5zZXJ0X2dldF9kYXRhYmFzZS5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla). Last update [c724ec7...14cba5c](https://codecov.io/gh/devanshshukla99/pytest-remote-response/pull/28?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Devansh+Shukla).


Patch:
diff --git a/docs/conf.py b/docs/conf.py
index 484ca87..ab1554b 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -260,5 +260,5 @@ intersphinx_mapping = {
     "urllib3": ("https://urllib3.readthedocs.io/en/stable/", None),
     "requests": ("https://docs.python-requests.org/en/master/", None),
     "aiohttp": ("https://docs.aiohttp.org/en/stable/", None),
-    "tinydb": ("https://tinydb.readthedocs.io/en/stable/", None),
+    "sqlite3": ("https://docs.python.org/3/library/sqlite3.html", None),
 }
diff --git a/docs/how_it_works.rst b/docs/how_it_works.rst
index ed72e7a..379dc26 100644
--- a/docs/how_it_works.rst
+++ b/docs/how_it_works.rst
@@ -83,7 +83,7 @@ Example
 📁Database
 ***********
 
-``pytest-remote-response`` has :class:`~pytest_response.database.ResponseDB` to facilitate talking to the database. Internally it uses :class:`~tinydb.database.TinyDB`.
+``pytest-remote-response`` has :class:`~pytest_response.database.ResponseDB` to facilitate talking to the database. Internally, its actually a wrapper for :mod:`sqlite3`.
 
 Primary communication methods are :meth:`~pytest_response.database.ResponseDB.insert` and :meth:`~pytest_response.database.ResponseDB.get`.
 
@@ -100,7 +100,7 @@ Primary communication methods are :meth:`~pytest_response.database.ResponseDB.in
       - Dumped as
 
     * - url
-      - :class:`str`
+      - :class:`str` -- :func:`base64.b64encode` serialized
 
     * - cache_date
       - :class:`str`
diff --git a/examples/insert_get_database.py b/examples/insert_get_database.py
index d013ce1..06e87ff 100644
--- a/examples/insert_get_database.py
+++ b/examples/insert_get_database.py
@@ -1,4 +1,6 @@
 # Import the database interface :class:`pytest_response.database.ResponseDB`
+from base64 import b64encode
+
 from pytest_response.database import ResponseDB
 
 # Setup the database
@@ -12,7 +14,7 @@ data = b""
 db.insert(url=url, status=status, response=data, headers=headers)
 
 # Verify the index
-assert url in db.index()
+assert b64encode(url.encode()).decode() in db.index()
 
 # Query for an URL
 url = "https://www.python.org"
diff --git a/examples/response_decorator_urllib3.py b/examples/response_decorator_urllib3.py
index 08a5de2..f4af2c2 100644
--- a/examples/response_decorator_urllib3.py
+++ b/examples/response_decorator_urllib3.py
@@ -1,4 +1,5 @@
 import urllib3  # noqa
+
 from pytest_response import response
 
 response.configure(remote=True, capture=True, response=False)
diff --git a/setup.cfg b/setup.cfg
index d2cb01e..809d86a 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -30,7 +30,6 @@ setup_requires =
 install_requires =
     pytest>=4.6
     requests>=2.25.1
-    tinydb
     aiohttp
 
 [options.entry_points]


Test patch:
diff --git a/pytest_response/app.py b/pytest_response/app.py
index dc477ed..86cc90c 100644
--- a/pytest_response/app.py
+++ b/pytest_response/app.py
@@ -110,7 +110,7 @@ class Response:
         remote: bool = False,
         response: bool = False,
         log_level: str = "debug",
-        database: str = "database.json",
+        database: str = "database.db",
     ) -> None:
 
         log.setLevel(log_level.upper())
diff --git a/pytest_response/database.py b/pytest_response/database.py
index 3ef8311..f4de556 100644
--- a/pytest_response/database.py
+++ b/pytest_response/database.py
@@ -1,20 +1,26 @@
 import ast
 import zlib
+import sqlite3
 from base64 import b64decode, b64encode
-from typing import List, Tuple
+from typing import List, Tuple, Optional
 from datetime import date
 from urllib.parse import urljoin, urlparse
 
-from tinydb import TinyDB, where
-
 from pytest_response.exceptions import MalformedUrl
 
 __all__ = ["ResponseDB"]
 
 
+def dict_factory(cursor, row):
+    d = {}
+    for idx, col in enumerate(cursor.description):
+        d[col[0]] = row[idx]
+    return d
+
+
 class ResponseDB:
     """
-    Basic database class for `pytest_response`
+    Wrapper class for sqlite3
 
     Parameters
     ----------
@@ -23,7 +29,7 @@ class ResponseDB:
 
     Examples
     --------
-    >>> db = ResponseDB("db.json")
+    >>> db = ResponseDB("database.db")
     """
 
     today = date.today().strftime("%Y-%m-%d")
@@ -32,13 +38,33 @@ class ResponseDB:
 
     def __init__(self, path: str) -> None:
         self._path = path
-        self._database = TinyDB(path)
+        self._database = sqlite3.connect(path)
+        self.setup()
         return
 
     def __repr__(self) -> str:
         return f"<database {self._path}>"
 
-    def index(self, index: str = "url") -> List[str]:
+    def setup(self) -> bool:
+        """
+        Function to setup the database table.
+        """
+        self._database.cursor().executescript(
+            """
+            CREATE TABLE IF NOT EXISTS records (
+            url TEXT PRIMARY KEY NOT NULL,
+            cache_date DATE NOT NULL,
+            status TEXT NOT NULL,
+            headers TEXT NOT NULL,
+            response TEXT NOT NULL
+        );
+        """
+        )
+        self._database.commit()
+        self._database.row_factory = dict_factory
+        return True
+
+    def index(self, index: Optional[str] = "url") -> List[str]:
         """
         Returns all occurances of the column `index`.
 
@@ -52,7 +78,7 @@ class ResponseDB:
         _occurances : `list`
             All occurances of the selected column `index`.
         """
-        elements = self._database.all()
+        elements = self._database.execute("SELECT * FROM records;").fetchall()
         _occurances = []
         for element in elements:
             _occurances.append(element.get(index))
@@ -68,6 +94,11 @@ class ResponseDB:
         ----------
         url : `str`
             URL to be sanatized.
+
+        Returns
+        -------
+        _url : `str`
+            Sanatized URL
         """
         try:
             _urlparsed = urlparse(url)
@@ -76,7 +107,7 @@ class ResponseDB:
         except Exception:
             raise MalformedUrl
 
-    def insert(self, url: str, response: bytes, headers: dict, status: int = 200, **kwargs) -> None:
+    def insert(self, url: str, response: bytes, headers: dict, status: Optional[int] = 200, **kwargs) -> None:
         """
         Method for dumping url, headers and responses to the database.
         All additonal kwargs are dumped as well.
@@ -91,15 +122,20 @@ class ResponseDB:
             Headers captured.
         status : `int`
             Status code of the response.
-        **kwargs : `dict`
+        kwargs : `dict`
             Any additional parameter to be dumped.
         """
-        kwargs.update({"url": self._sanatize_url(url)})
-        kwargs.update({"cache_date": self.today})
-        kwargs.update({"status": str(status)})
-        kwargs.update({"headers": b64encode(zlib.compress(str(headers).encode("utf-8"))).decode("utf-8")})
-        kwargs.update({"response": b64encode(zlib.compress(response)).decode("utf-8")})
-        self._database.upsert(kwargs, where("url") == url)
+        _kwargs = [
+            b64encode(self._sanatize_url(url).encode()).decode(),
+            self.today,
+            str(status),
+            b64encode(zlib.compress(str(headers).encode("utf-8"))).decode("utf-8"),
+            b64encode(zlib.compress(response)).decode("utf-8"),
+        ]
+        self._database.execute(
+            "REPLACE INTO records(url, cache_date, status, headers, response) VALUES (?,?,?,?,?)", _kwargs
+        )
+        self._database.commit()
         return
 
     def get(self, url: str, **kwargs) -> Tuple[int, bytes, dict]:
@@ -121,8 +157,8 @@ class ResponseDB:
         headers : `dict`
             Response header.
         """
-        query = where("url") == self._sanatize_url(url)  # and where("request") == "req"
-        element = self._database.search(query)
+        url = b64encode(self._sanatize_url(url).encode()).decode()
+        element = self._database.execute(f"SELECT * FROM records WHERE url='{url}';").fetchall()
         if element:
             status = element[-1].get("status", 200)
             headers = element[-1].get("headers", "[]")
@@ -142,23 +178,21 @@ class ResponseDB:
         -------
         Return list of all elements.
         """
-        return self._database.all()
+        return self._database.execute("SELECT * FROM records").fetchall()
 
-    def truncate(self):
+    def truncate(self) -> bool:
         """
         Method to purge all records in the database.
         """
-        return self._database.truncate()
+        self._database.execute("DELETE FROM records;")
+        return self._database.commit()
 
     def close(self) -> None:
-        if isinstance(self._database, TinyDB):
-            if self._database._opened:
-                self._database.close()
-
-    def __del__(self):
-        if isinstance(self._database, TinyDB):
-            if self._database._opened:
-                self._database.close()
+        self._database.close()
+        return
+
+    def __del__(self) -> None:
+        self._database.close()
         return
 
     pass
diff --git a/pytest_response/plugin.py b/pytest_response/plugin.py
index 6bcc540..045a404 100644
--- a/pytest_response/plugin.py
+++ b/pytest_response/plugin.py
@@ -26,7 +26,7 @@ def pytest_addoption(parser):
         dest="remote_db",
         action="store",
         type=str,
-        default="database.json",
+        default="database.db",
         help="Filename to store and mock the connections requests --remote-database=[DUMPFILE]",
     )
     parser.addoption(
diff --git a/tests/test_aiohttp.py b/tests/test_aiohttp.py
index cf6177e..6fe8fe9 100644
--- a/tests/test_aiohttp.py
+++ b/tests/test_aiohttp.py
@@ -4,9 +4,11 @@ import pytest
 @pytest.fixture
 def testcode():
     return """
-        import aiohttp
         import pytest
+        import aiohttp
         import asyncio
+        from base64 import b64encode
+
         from pytest_response import response
 
         @pytest.fixture(scope="function")
@@ -40,8 +42,8 @@ def testcode():
             event_loop.run_until_complete(async_whats_this())
 
         def test_database():
-            assert response.db.index() == ["http://www.testingmcafeesites.com/testcat_ac.html",
-                                           "https://www.python.org"]
+            assert response.db.index() == [b64encode(b"http://www.testingmcafeesites.com/testcat_ac.html").decode(),
+                                           b64encode(b"https://www.python.org").decode()]
         """
 
 
diff --git a/tests/test_examples.py b/tests/test_examples.py
index d261397..1b313e0 100644
--- a/tests/test_examples.py
+++ b/tests/test_examples.py
@@ -2,6 +2,8 @@ def test_examples(pytester):
     pytester.makepyfile(
         """
         import pytest
+        from base64 import b64encode
+
         from pytest_response import response
         from pytest_response.exceptions import RemoteBlockedError
 
@@ -12,7 +14,7 @@ def test_examples(pytester):
 
         def test_capture_requests():
             from examples import capture_requests  # noqa
-            assert response.db.index() == ["https://www.python.org"]
+            assert response.db.index() == [b64encode(b"https://www.python.org").decode()]
             return
 
         def test_response_urllib3():
diff --git a/tests/test_requests.py b/tests/test_requests.py
index 0a4dd00..3e76f15 100644
--- a/tests/test_requests.py
+++ b/tests/test_requests.py
@@ -5,6 +5,8 @@ import pytest
 def testcode():
     return """
         import requests
+        from base64 import b64encode
+
         from pytest_response import response
 
         @response.activate("requests")
@@ -22,8 +24,8 @@ def testcode():
             assert res.content
 
         def test_database():
-            assert response.db.index() == ["http://www.testingmcafeesites.com/testcat_ac.html",
-                                           "https://www.python.org"]
+            assert response.db.index() == [b64encode(b"http://www.testingmcafeesites.com/testcat_ac.html").decode(),
+                                           b64encode(b"https://www.python.org").decode()]
         """
 
 
diff --git a/tests/test_urllib.py b/tests/test_urllib.py
index a73e3fc..c806ba8 100644
--- a/tests/test_urllib.py
+++ b/tests/test_urllib.py
@@ -5,6 +5,8 @@ import pytest
 def testcode():
     return """
         import urllib.request
+        from base64 import b64encode
+
         from pytest_response import response
 
         response.unapply()
@@ -24,8 +26,8 @@ def testcode():
             assert res.read()
 
         def test_database():
-            assert response.db.index() == ["http://www.testingmcafeesites.com/testcat_ac.html",
-                                           "https://www.python.org"]
+            assert response.db.index() == [b64encode(b"http://www.testingmcafeesites.com/testcat_ac.html").decode(),
+                                           b64encode(b"https://www.python.org").decode()]
         """
 
 
diff --git a/tests/test_urllib3.py b/tests/test_urllib3.py
index c70ddf5..cd97178 100644
--- a/tests/test_urllib3.py
+++ b/tests/test_urllib3.py
@@ -1,4 +1,5 @@
 import pytest
+
 from pytest_response.logger import log
 
 
@@ -6,6 +7,8 @@ from pytest_response.logger import log
 def testcode():
     return """
         import urllib3
+        from base64 import b64encode
+
         from pytest_response import response
 
         @response.activate("urllib3")
@@ -25,8 +28,8 @@ def testcode():
             assert res.data
 
         def test_database():
-            assert response.db.index() == ["http://www.testingmcafeesites.com/testcat_ac.html",
-                                           "https://www.python.org"]
+            assert response.db.index() == [b64encode(b"http://www.testingmcafeesites.com/testcat_ac.html").decode(),
+                                           b64encode(b"https://www.python.org").decode()]
         """