Repository: sec-edgar/sec-edgar

Issue:
Missing fillings on request, provided cik, information gaps
**Describe the bug**
Using the generic code you provide, I'm requesting 10-k and 10-q filings. Unfortunately I'm only receiving information up to 2009 (where there is a format change but the filing is consistent), and missing years 2016. 

**To Reproduce**
Steps to reproduce the behaviour:

```python
company_ciks = {
    'alphabet': ['0001288776', '0001652044']
}

dir_path = '_dataset/'

requests_counter = 0
for company_name, ciks in company_ciks.items():
    company_path = dir_path + company_name
    
    for cik in ciks:
        requests_counter += 1

        my_filings = Filing(cik_lookup=cik, filing_type=FilingType.FILING_10Q)
        my_filings.save(company_path)
        
        print('\rCompleted ' + str(requests_counter) + ' requests...', end='')
        sleep(1)

print('\nObtained all 10Q filings...')
```


I've attached my code to demonstrate how my fillings are being pulled. Just wrapped in some loops nothing fancy. Yet we still miss information that is available.

Here are the filings for the first cik, showing them going back to 2005.
https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001288776&type=10-q&dateb=&owner=exclude&count=40

I have a feeling it could be timed out information, I'm going to try pull all the data again to see. I'm aware edgar has a 10 request per second limit which might cause some throttling and timeouts.

Really love sec-edgar, keep up the great work! :)


Discussion:

Patch:
diff --git a/docs/source/whatsnew.rst b/docs/source/whatsnew.rst
index 3384487..abe61cb 100644
--- a/docs/source/whatsnew.rst
+++ b/docs/source/whatsnew.rst
@@ -6,6 +6,8 @@ What's New
 
 New features, bug fixes, and improvements for each release.
 
+.. include:: whatsnew/v0.1.7.rst
+
 .. include:: whatsnew/v0.1.6.rst
 
 .. include:: whatsnew/v0.1.5.rst
diff --git a/docs/source/whatsnew/v0.1.1.rst b/docs/source/whatsnew/v0.1.1.rst
index 6207ae9..3824dfb 100644
--- a/docs/source/whatsnew/v0.1.1.rst
+++ b/docs/source/whatsnew/v0.1.1.rst
@@ -7,4 +7,10 @@ Highlights
 ~~~~~~~~~~
 
 * Add API for 10-K
-* Create a central API point using class
\ No newline at end of file
+* Create a central API point using class
+
+Contributors
+~~~~~~~~~~~~
+
+- npsolve
+- zevaverbach
\ No newline at end of file
diff --git a/docs/source/whatsnew/v0.1.2.rst b/docs/source/whatsnew/v0.1.2.rst
index d9f6315..405efad 100644
--- a/docs/source/whatsnew/v0.1.2.rst
+++ b/docs/source/whatsnew/v0.1.2.rst
@@ -7,4 +7,10 @@ Highlights
 ~~~~~~~~~~
 
 * Fix issue related to file saving
-* Remove platform dependent file path
\ No newline at end of file
+* Remove platform dependent file path
+
+Contributors
+~~~~~~~~~~~~
+
+- npsolve
+- Quartzing
\ No newline at end of file
diff --git a/docs/source/whatsnew/v0.1.3.rst b/docs/source/whatsnew/v0.1.3.rst
index 1c1915c..5dc8f7c 100644
--- a/docs/source/whatsnew/v0.1.3.rst
+++ b/docs/source/whatsnew/v0.1.3.rst
@@ -7,4 +7,13 @@ Highlights
 ~~~~~~~~~~
 
 * Add support for SD forms
-* Fixed bugs for python3
\ No newline at end of file
+* Fixed bugs for python3
+
+Contributors
+~~~~~~~~~~~~
+
+- crperezt
+- franciosi
+- jackmoody11
+- jblemoine
+- npsolve
\ No newline at end of file
diff --git a/docs/source/whatsnew/v0.1.4.rst b/docs/source/whatsnew/v0.1.4.rst
index 9f8d7ed..a874f3f 100644
--- a/docs/source/whatsnew/v0.1.4.rst
+++ b/docs/source/whatsnew/v0.1.4.rst
@@ -8,4 +8,10 @@ Highlights
 
 * Refactor code to have a common method for fetching filings
 * Provide user to add their own repo path where filings can be downloaded
-* Add CI pipeline using Travis
\ No newline at end of file
+* Add CI pipeline using Travis
+
+Contributors
+~~~~~~~~~~~~
+
+- jackmoody11
+- johnisanerd
\ No newline at end of file
diff --git a/docs/source/whatsnew/v0.1.5.rst b/docs/source/whatsnew/v0.1.5.rst
index 04c220a..285e25f 100644
--- a/docs/source/whatsnew/v0.1.5.rst
+++ b/docs/source/whatsnew/v0.1.5.rst
@@ -32,4 +32,9 @@ Bugs
 Enhancements
 ~~~~~~~~~~~~
 
-* Introduce :class:`secedgar.filings.base.AbstractFiling` class as abstract base class
\ No newline at end of file
+* Introduce :class:`secedgar.filings.base.AbstractFiling` class as abstract base class
+
+Contributors
+~~~~~~~~~~~~
+
+- jackmoody11
\ No newline at end of file
diff --git a/docs/source/whatsnew/v0.1.6.rst b/docs/source/whatsnew/v0.1.6.rst
index e2a02f5..6e0dd60 100644
--- a/docs/source/whatsnew/v0.1.6.rst
+++ b/docs/source/whatsnew/v0.1.6.rst
@@ -5,4 +5,10 @@ Highlights
 ~~~~~~~~~~
 
 * :class:`secedgar.filings.DailyFilings` added to allow easy access to every day's filings.
-* :meth:`secedgar.utils.get_cik_map` added to add an easy and fast option for getting company CIKs.
\ No newline at end of file
+* :meth:`secedgar.utils.get_cik_map` added to add an easy and fast option for getting company CIKs.
+
+Contributors
+~~~~~~~~~~~~
+
+- jackmoody11
+- tomfunk
\ No newline at end of file
diff --git a/docs/source/whatsnew/v0.1.7.rst b/docs/source/whatsnew/v0.1.7.rst
index 0826855..2ccc0e2 100644
--- a/docs/source/whatsnew/v0.1.7.rst
+++ b/docs/source/whatsnew/v0.1.7.rst
@@ -5,8 +5,11 @@ Highlights
 ~~~~~~~~~~
 
 * Fixes `DailyFilings` bug to properly fetch daily filings.
+* Fixes `Filing` bug to fetch specified number of filings.
 
 Contributors
 ~~~~~~~~~~~~
 
-- jonzd
\ No newline at end of file
+- jackmoody11
+- jonzd
+- Marsh-James
\ No newline at end of file
diff --git a/secedgar/client/_base.py b/secedgar/client/_base.py
index 89467a3..9e967da 100644
--- a/secedgar/client/_base.py
+++ b/secedgar/client/_base.py
@@ -37,6 +37,6 @@ class AbstractClient(ABC):
 
     @property
     @abstractmethod
-    def count(self):
+    def batch_size(self):
         """Number of results per page searched. Increasing can improve speed for large requests."""
         pass  # pragma: no cover
diff --git a/secedgar/client/network_client.py b/secedgar/client/network_client.py
index 7516bf2..89f92ac 100644
--- a/secedgar/client/network_client.py
+++ b/secedgar/client/network_client.py
@@ -14,7 +14,7 @@ class NetworkClient(AbstractClient):
     Attributes:
         retry_count (int): Number of times to retry connecting to URL if not successful.
         pause (float): Time (in seconds) to wait before retrying if not successful.
-        count (int): Number of filings to receive (helpful if pagination needed).
+        batch_size (int): Number of filings to receive per request (helpful if pagination needed).
     """
 
     _BASE = "http://www.sec.gov/"
@@ -22,7 +22,7 @@ class NetworkClient(AbstractClient):
     def __init__(self, **kwargs):
         self.retry_count = kwargs.get("retry_count", 3)
         self.pause = kwargs.get("pause", 0.5)
-        self.count = kwargs.get("count", 10)
+        self.batch_size = kwargs.get("batch_size", 10)
         self.response = None
 
     @property
@@ -52,17 +52,17 @@ class NetworkClient(AbstractClient):
         self._pause = value
 
     @property
-    def count(self):
+    def batch_size(self):
         """The Number of results to show per page."""
-        return self._count
+        return self._batch_size
 
-    @count.setter
-    def count(self, value):
+    @batch_size.setter
+    def batch_size(self, value):
         if not isinstance(value, int):
-            raise TypeError("Count must be int. Given type {0}".format(type(value)))
+            raise TypeError("Batch size must be int. Given type {0}".format(type(value)))
         elif value < 1:
-            raise ValueError("Count must be positive integer.")
-        self._count = value
+            raise ValueError("Batch size must be positive integer.")
+        self._batch_size = value
 
     @staticmethod
     def _prepare_query(path):
diff --git a/secedgar/filings/daily.py b/secedgar/filings/daily.py
index b11d124..deac90e 100644
--- a/secedgar/filings/daily.py
+++ b/secedgar/filings/daily.py
@@ -99,7 +99,8 @@ class DailyFilings(AbstractFiling):
         Args:
             update_cache (bool, optional): Whether master index should be updated
                 method call. Defaults to False.
-            kwargs: Keyword arguments to pass to `client.get_response`.
+            kwargs: Keyword arguments to pass to
+                ``secedgar.client._base.AbstractClient.get_response``.
 
         Returns:
             text (str): Idx file text.
@@ -122,7 +123,7 @@ class DailyFilings(AbstractFiling):
                     date=formatted_date))
         return self._master_idx_file
 
-    def get_paths(self, update_cache=False, **kwargs):
+    def get_paths(self, update_cache=False):
         """Gets all paths for given day.
 
         Each path will look something like
@@ -150,6 +151,8 @@ class DailyFilings(AbstractFiling):
         Args:
             update_cache (bool, optional): Whether filings dict should be
                 updated on each method call. Defaults to False.
+            kwargs: Any kwargs to pass to _get_master_idx_file. See
+                ``secedgar.filings.daily.DailyFilings._get_master_idx_file``.
         """
         if self._filings_dict is None or update_cache:
             idx_file = self._get_master_idx_file(**kwargs)
diff --git a/secedgar/filings/filing.py b/secedgar/filings/filing.py
index 0056c6b..861f0f2 100644
--- a/secedgar/filings/filing.py
+++ b/secedgar/filings/filing.py
@@ -22,10 +22,9 @@ class Filing(AbstractFiling):
             Defaults to None (will fetch all filings before end_date).
         end_date (Union[str, datetime.datetime], optional): Date after which not to fetch reports.
             Stands for "date before." Defaults to today.
-
-    Keyword Args:
-        count (int): Number of filings to get per request. Passed to NetworkClient
-            created on initialization if none given.
+        count (int): Number of filings to fetch. Will fetch up to `count` if that many filings
+            are available. Defaults to all filings available.
+        kwargs: See kwargs accepted for :class:`secedgar.client.network_client.NetworkClient`.
 
     .. versionadded:: 0.1.5
     """
@@ -36,6 +35,7 @@ class Filing(AbstractFiling):
                  start_date=None,
                  end_date=datetime.datetime.today(),
                  client=None,
+                 count=None,
                  **kwargs):
         self._start_date = start_date
         self._end_date = end_date
@@ -55,8 +55,9 @@ class Filing(AbstractFiling):
             'start': 0,
             'type': self.filing_type.value
         }
-        if kwargs.get('count') is not None:
-            self._params['count'] = kwargs.get('count')
+        self._count = count
+        if count is not None:
+            self._params['count'] = count
         if start_date is not None:
             self._params['datea'] = sanitize_date(start_date)
         # Make default client NetworkClient and pass in kwargs
@@ -109,6 +110,18 @@ class Filing(AbstractFiling):
         self._filing_type = filing_type
         self._params['type'] = filing_type.value
 
+    @property
+    def count(self):
+        """Number of filings to fetch."""
+        return self._count
+
+    @count.setter
+    def count(self, val):
+        if not (val is None or isinstance(val, int)) or val < 1:
+            raise TypeError("Count must be positive integer or None.")
+        self._count = val
+        self._params['count'] = val
+
     @property
     def accession_numbers(self):
         """List of accession numbers for filings."""
@@ -123,10 +136,9 @@ class Filing(AbstractFiling):
         """Get urls for all CIKs given to Filing object.
 
         Args:
-            kwargs: Anything to be passed to requests when making get request.
-
-        Keyword Args:
-            See keyword arguments accepted for ``secedgar.client._base.AbstractClient.get_soup``
+            **kwargs: Anything to be passed to requests when making get request.
+                See keyword arguments accepted for
+                ``secedgar.client._base.AbstractClient.get_soup``.
 
         Returns:
             urls (list): List of urls for txt files to download.
@@ -144,10 +156,9 @@ class Filing(AbstractFiling):
 
         Args:
             cik (str): CIK for company.
-            kwargs: Anything to be passed to requests when making get request.
-
-        Keyword Args:
-            See keyword arguments accepted for ``secedgar.client._base.AbstractClient.get_soup``
+            **kwargs: Anything to be passed to requests when making get request.
+                See keyword arguments accepted for
+                ``secedgar.client._base.AbstractClient.get_soup``.
 
         Returns:
             txt_urls (list of str): Up to the desired number of URLs for that specific company
@@ -158,15 +169,16 @@ class Filing(AbstractFiling):
         self.params["start"] = 0  # set start back to 0 before paginating
 
         # TODO: Make paginate utility outside of this class
-        while len(links) < self._client.count:
-            data = self._client.get_soup(self.path, self.params, **kwargs)
+        while self.count is None or len(links) < self.count:
+            data = self.client.get_soup(self.path, self.params, **kwargs)
             links.extend([link.string for link in data.find_all("filinghref")])
-            self.params["start"] += self._client.count
-            if len(data.find_all("filinghref")) == 0:
-                break  # break if no more filings left
+            self.params["start"] += self.client.batch_size
+            if len(data.find_all("filinghref")) == 0:  # no more filings
+                break
 
         txt_urls = [link[:link.rfind("-")].strip() + ".txt" for link in links]
-        return txt_urls[:self.client.count]
+        # Takes `count` filings at most
+        return txt_urls[:self.count]
 
     def _get_accession_numbers(self, links):
         """Gets accession numbers given list of links.
diff --git a/setup.py b/setup.py
index f403349..8550efd 100644
--- a/setup.py
+++ b/setup.py
@@ -60,9 +60,9 @@ setup(
                  for automatically collecting public filings from the SEC.""",  # noqa: W291
     long_description=LONG_DESCRIPTION,
     long_description_content_type='text/x-rst',
-    entry_points='''
-            [console_scripts]
-            ''',
+    entry_points={
+        'console_scripts': []
+    },
     cmdclass={
         'install': install,
     },


Test patch:
diff --git a/secedgar/tests/client/test_network_client.py b/secedgar/tests/client/test_network_client.py
index ce6bd1d..fb67738 100644
--- a/secedgar/tests/client/test_network_client.py
+++ b/secedgar/tests/client/test_network_client.py
@@ -129,11 +129,11 @@ class TestNetworkClient:
     @pytest.mark.parametrize(
         "test_input,expectation",
         [
-            (0.5, pytest.raises(TypeError)),
-            ("1", pytest.raises(TypeError)),
-            (0, pytest.raises(ValueError))
+            (0.5, TypeError),
+            ("1", TypeError),
+            (0, ValueError)
         ]
     )
-    def test_client_bad_count_setter(self, test_input, expectation, client):
-        with expectation:
-            client.count = test_input
+    def test_client_bad_batch_size_setter(self, test_input, expectation, client):
+        with pytest.raises(expectation):
+            client.batch_size = test_input
diff --git a/secedgar/tests/filings/test_filings.py b/secedgar/tests/filings/test_filings.py
index e50d698..41ca95e 100644
--- a/secedgar/tests/filings/test_filings.py
+++ b/secedgar/tests/filings/test_filings.py
@@ -3,7 +3,7 @@ import datetime
 import pytest
 
 from secedgar.client import NetworkClient
-from secedgar.filings import Filing, FilingType, CIKLookup
+from secedgar.filings import Filing, FilingType
 
 from secedgar.filings.cik_validator import _CIKValidator
 from secedgar.tests.utils import datapath
@@ -45,14 +45,15 @@ class MockCIKValidatorMultipleCIKs:
 class TestFiling(object):
     @pytest.mark.slow
     def test_count_returns_exact(self, monkeypatch):
-        aapl = Filing(cik_lookup='aapl', filing_type=FilingType.FILING_10Q, count=10)
+        count = 10
+        aapl = Filing(cik_lookup='aapl', filing_type=FilingType.FILING_10Q, count=count)
         monkeypatch.setattr(_CIKValidator, "get_ciks", MockCIKValidatorGetCIKs.get_ciks)
         monkeypatch.setattr(NetworkClient, "get_response", MockSingleCIKFiling)
         urls = aapl.get_urls()['aapl']
-        if len(urls) != aapl.client.count:
+        if len(urls) != count:
             raise AssertionError("""Count should return exact number of filings.
                                  Got {0}, but expected {1} URLs.""".format(
-                urls, aapl.client.count))
+                urls, count))
 
     def test_date_is_sanitized(self):
         start_date = datetime.datetime(2012, 3, 1)
@@ -99,10 +100,6 @@ class TestFiling(object):
         first_txt_url = aapl.get_urls()['aapl'][0]
         assert first_txt_url.split('.')[-1] == 'txt'
 
-    def test_invalid_filing_type_enum(self):
-        with pytest.raises(AttributeError):
-            Filing(cik_lookup='0000320193', filing_type=FilingType.INVALID)
-
     @pytest.mark.parametrize(
         "new_filing_type",
         (
@@ -192,6 +189,21 @@ class TestFiling(object):
         monkeypatch.setattr(_CIKValidator, "get_ciks", MockCIKValidatorMultipleCIKs.get_ciks)
         # Use same response for each request
         monkeypatch.setattr(NetworkClient, "get_response", MockSingleCIKFiling)
-        ciks = CIKLookup(['aapl', 'msft', 'amzn'])
-        f = Filing(ciks, FilingType.FILING_10Q, count=3)
-        assert all(len(f.get_urls().get(key)) == 3 for key in f.get_urls().keys())
+        f = Filing(cik_lookup=['aapl', 'msft', 'amzn'], filing_type=FilingType.FILING_10Q, count=5)
+        assert all(len(f.get_urls().get(key)) == 5 for key in f.get_urls().keys())
+
+    @pytest.mark.parametrize(
+        "count",
+        [
+            10,
+            25,
+            30
+        ]
+    )
+    def test_filing_returns_correct_number_of_urls(self, monkeypatch, count):
+        monkeypatch.setattr(_CIKValidator, "get_ciks", MockCIKValidatorMultipleCIKs.get_ciks)
+        # Use same response for each request
+        monkeypatch.setattr(NetworkClient, "get_response", MockSingleCIKFiling)
+        f = Filing(cik_lookup=['aapl', 'msft', 'amzn'], filing_type=FilingType.FILING_10Q,
+                   count=count, client=NetworkClient(batch_size=10))
+        assert all(len(f.get_urls().get(key)) == count for key in f.get_urls().keys())