Repository: sec-edgar/sec-edgar

Issue:
Datetime Error on Daily Fillings
**Describe the bug**
`TypeError: can't compare datetime.datetime to datetime.date.`

**To Reproduce**
Steps to reproduce the behavior:
1. `$ secedgar daily -d 20210203`
2. See error
```
    elif self._date < datetime.date(1998, 3, 31):
TypeError: can't compare datetime.datetime to datetime.date
```

**Expected behavior**
Download daily fillings

**Desktop (please complete the following information):**
```
Arch Linux 5.10.7-arch1-1
Python 3.9.1

Package           Version
----------------- ---------
aiohttp           3.7.3
async-timeout     3.0.1
attrs             20.3.0
beautifulsoup4    4.9.3
certifi           2020.12.5
chardet           3.0.4
click             7.1.2
idna              2.10
lxml              4.6.2
multidict         5.1.0
pip               20.3.3
requests          2.25.1
secedgar          0.3.2
setuptools        51.0.0
soupsieve         2.1
tqdm              4.56.0
typing-extensions 3.7.4.3
urllib3           1.26.3
wheel             0.36.2
yarl              1.6.3


```


Discussion:
lolski: Still doesn't work as per commit `c05e06d824aa3df3d9334835fbb3bd7d6f6c7fe2` I think:

<img width="1240" alt="Screen Shot 2021-04-30 at 8 21 47 AM" src="https://user-images.githubusercontent.com/464164/116662023-27507800-a98d-11eb-8298-0b992c7069ae.png">

jackmoody11: I will see if I can figure this out. Thanks @lolski.
sroussey: Is this the reason behind this error:

```Traceback (most recent call last):
  File "/usr/local/bin/secedgar", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/secedgar/cli.py", line 75, in daily
    d = DailyFilings(date=date_cleanup(date))
  File "/usr/local/lib/python3.9/site-packages/secedgar/cli.py", line 26, in date_cleanup
    return datetime.strptime(date, "%Y%m%d") if date is not None else None
  File "/usr/local/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/local/lib/python3.9/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '2021-07-23' does not match format '%Y%m%d'
~/C/e/secedgar ❯❯❯ docker run -it --mount src="$(pwd)/filings",target=/code/filings,type=bind sec-edgar daily "--directory" /code/filings "--date" "20210723"

Traceback (most recent call last):
  File "/usr/local/bin/secedgar", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/secedgar/cli.py", line 76, in daily
    d.save(directory=directory)
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/daily.py", line 137, in save
    self._save_filings(directory,
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/_index.py", line 295, in _save_filings
    urls = self._check_urls_exist()
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/_base.py", line 98, in _check_urls_exist
    urls = self.get_urls()
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/_index.py", line 170, in get_urls
    filings_dict = self.get_filings_dict()
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/_index.py", line 140, in get_filings_dict
    idx_file = self._get_master_idx_file(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/_index.py", line 118, in _get_master_idx_file
    if self.idx_filename in self._get_listings_directory().text:
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/daily.py", line 84, in idx_filename
    return "master.{date}.idx".format(date=self._get_idx_formatted_date())
  File "/usr/local/lib/python3.9/site-packages/secedgar/filings/daily.py", line 104, in _get_idx_formatted_date
    elif self._date < datetime.date(1998, 3, 31):
TypeError: can't compare datetime.datetime to datetime.date```
jackmoody11: This PR should solve the issue. Finalizing now to get this into v0.4.0-alpha.2.

Patch:
diff --git a/docs/source/whatsnew/v0.4.0.rst b/docs/source/whatsnew/v0.4.0.rst
index 81f317a..27eacc0 100644
--- a/docs/source/whatsnew/v0.4.0.rst
+++ b/docs/source/whatsnew/v0.4.0.rst
@@ -14,9 +14,11 @@ Many breaking changes, as this update does a lot of restructuring.
 - Add many more types to ``FilingType`` enum. Switch to csv table for details.
 - If no filings found for ``DailyFilings`` or ``MasterFilings`` after finding that other filings that do not match the criteria exist, then raise a ``NoFilingsError`` instead of a ``ValueError``.
 - Fix use of ``user_agent`` by ``NetworkClient`` and require its use by all filings classes/functions
+- Fix issue where daily CLI command would not work due to ``date_cleanup`` returning ``datetime.datetime`` instead of ``datetime.date``
 
 Contributors
 ~~~~~~~~~~~~
 
 - reteps
-- jackmoody11
\ No newline at end of file
+- jackmoody11
+- lolski
\ No newline at end of file
diff --git a/secedgar/__init__.py b/secedgar/__init__.py
index 61a6fdf..cad79a0 100644
--- a/secedgar/__init__.py
+++ b/secedgar/__init__.py
@@ -1,4 +1,4 @@
-__version__ = '0.4.0-alpha.1'
+__version__ = '0.4.0-alpha.2'
 
 from secedgar.core.combo import ComboFilings  # noqa: F401
 from secedgar.core.company import CompanyFilings  # noqa: F401
diff --git a/secedgar/cli.py b/secedgar/cli.py
index f1f53a4..e343b5b 100644
--- a/secedgar/cli.py
+++ b/secedgar/cli.py
@@ -14,9 +14,7 @@ from secedgar.exceptions import FilingTypeError
               type=str)
 @click.pass_context
 def cli(ctx, user_agent):
-    r"""Main CLI group.
-
-    \f
+    """Main CLI group.
 
     Args:
         ctx (click.core.Context): Click context.
@@ -30,16 +28,16 @@ def cli(ctx, user_agent):
 
 
 def date_cleanup(date):
-    r"""Transforms date of form YYYYMMDD to datetime object.
+    """Transforms date of form YYYYMMDD to datetime object.
 
     Args:
-        date (Union[str, NoneType]): Date of the form YYYYMMDD to be transformed.
+        date (str): Date of the form YYYYMMDD to be transformed.
 
     Returns:
-        ``datetime.datetime`` object if given string.
-        Returns None if None is given.
+        ``datetime.date`` object if given string.
+        If given None, None is returned.
     """
-    return datetime.strptime(date, "%Y%m%d") if date is not None else None
+    return datetime.strptime(date, "%Y%m%d").date() if date is not None else None
 
 
 @cli.command()
@@ -71,9 +69,7 @@ def date_cleanup(date):
               default=os.getcwd(), type=str)
 @click.pass_context
 def filing(ctx, lookups, ftype, start, end, count, directory):
-    r"""Click command for downloading filings. Run ``secedgar filing --help`` for info.
-
-    \f
+    """Click command for downloading filings. Run ``secedgar filing --help`` for info.
 
     Args:
         ctx (click.core.Context): Click context.
@@ -118,6 +114,13 @@ def filing(ctx, lookups, ftype, start, end, count, directory):
               default=os.getcwd(), type=str)
 @click.pass_context
 def daily(ctx, date, directory):
-    """Click command for downloading daily filings. Run ``secedgar daily --help`` for info."""
+    """Click command for downloading daily filings. Run ``secedgar daily --help`` for info.
+
+    Args:
+        ctx (click.core.Context): Click context.
+        date (str): Date to look up daily filings for. Should be in the format YYYYMMDD.
+        directory (str): Directory where files should be saved.
+            Defaults to current working directory.
+    """
     d = DailyFilings(date=date_cleanup(date), user_agent=ctx.obj['user_agent'])
     d.save(directory=directory)


Test patch:
diff --git a/secedgar/tests/test_cli.py b/secedgar/tests/test_cli.py
index e69647a..7dac491 100644
--- a/secedgar/tests/test_cli.py
+++ b/secedgar/tests/test_cli.py
@@ -1,8 +1,10 @@
+import datetime
 import os
+from datetime import date
 
 import pytest
 from click.testing import CliRunner
-from secedgar.cli import cli
+from secedgar.cli import cli, date_cleanup
 from secedgar.exceptions import FilingTypeError
 # Borrow mocks without including in conftest
 from secedgar.tests.core.test_daily import mock_daily_idx_file  # noqa: F401
@@ -104,3 +106,21 @@ class TestCLI:
     )
     def test_cli_requires_user_agent(self, user_input, tmp_data_directory):
         check_bad_inputs(cli, user_input, SystemExit, tmp_data_directory, user_agent=None)
+
+    @pytest.mark.parametrize(
+        "input,output",
+        [
+            ("20210423", date(2021, 4, 23)),
+            ("19900101", date(1990, 1, 1)),
+            ("20000101", date(2000, 1, 1)),
+            (None, None)
+        ]
+
+    )
+    def test_date_cleanup(self, input, output):
+        result = date_cleanup(input)
+        if isinstance(input, str):
+            assert isinstance(result, datetime.date)
+        else:
+            assert result is None
+        assert result == output