Repository: biglocalnews/court-scraper (language: python)

Issue:
Odyssey ResultRow is overly restrictive
The code for [identifying cases listed on search result pages][] is overly restrictive due to assumptions about the format of case numbers.

## Fix

Delete [`ResultRow.is_case_row`][] and modify related `_prep_case_rows` as below:

```
      def _prep_case_rows(self, elements):
          case_rows = []
          for el in elements:
~             #row = ResultRow(el)
~             #if row.is_case_row:
+             #    case_rows.append(row)
~             case_rows.append(ResultRow(el))
          return case_rows

```

[identifying cases listed on search result pages]: https://github.com/biglocalnews/court-scraper/blob/14e5180c8825f6d19c27eee53e54b70b276ee924/court_scraper/platforms/odyssey_site/pages/search_results.py#L55

[`ResultRow.is_case_row`]: https://github.com/biglocalnews/court-scraper/blob/14e5180c8825f6d19c27eee53e54b70b276ee924/court_scraper/platforms/odyssey_site/pages/search_results.py#L83
Add pytest runslow option for live tests on Odyssey
> OdysseySite "live" tests will typically require a user/pass 

Testing against live services is gross, but some JS-heavy platforms such as Odyssey are torturous to test otherwise.

Start building out a test suite with support for a pytest [`runslow`](https://docs.pytest.org/en/stable/example/simple.html?highlight=skip#control-skipping-of-tests-according-to-command-line-option) CLI flag and test mark that allows us to run "slow" (read live) tests when needed to avoid regressions.

Start building out the test suite while working on #49, #51 and #50
OdysseySite.search times out when extracting case details
[OdysseySite.search][] times out when extracting case details on pages without `Financials` section. Need a better strategy for ensuring page load completion since Financials section isn't always present.

[OdysseySite.search]: https://github.com/biglocalnews/court-scraper/blob/master/court_scraper/platforms/odyssey_site/site.py#L31

## To reproduce

> Fill in actual user/pass below

```python
from court_scraper.platforms.odyssey_site import OdysseySite

url = "https://cmsportal.chathamcounty.org/Portal/Home/Dashboard/29"
user = 'USER'
password = 'PASSWORD'
site = OdysseySite(url, user, password, download_dir='/tmp')
site.login(headless=False)
results = site.search(search_terms=["MGCV20-00001"], headless=False)
```

## Stacktrace

```
Traceback (most recent call last):
  File "ga_chatham_indexer.py", line 42, in <module>
    main()
  File "ga_chatham_indexer.py", line 18, in main
    results = site.search(search_terms=search_terms, headless=False)
  File "/Users/tumgoren/code/stanford/biglocal/court-scraper/court_scraper/platforms/odyssey_site/site.py", line 47, in search
    row_data['page_source'] = detail_page.page_source
  File "/Users/tumgoren/code/stanford/biglocal/court-scraper/court_scraper/platforms/odyssey_site/pages/case_detail.py", line 24, in page_source
    'Financial'
  File "/Users/tumgoren/.pyenv/versions/test-court-scraper/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

```



Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index e2458d2..8c7aee3 100644
--- a/README.md
+++ b/README.md
@@ -4,6 +4,7 @@
 - [Setup](#setup)
 - [Usage](#usage)
 - [Contributing](#contributing)
+- [Testing](#testing)
 
 ## Overview
 
@@ -246,10 +247,10 @@ platform. For example, there are separate entries in [sites_meta.csv][]
 for Chatham, Dekalb and Fulton counties in Georgia, even though all
 three sites use the same version of the Odyssey courts platform.
 
-### Testing
+## Testing
 
 This code base was developed on Python 3.7 and uses the
-[pytest](https://docs.pytest.org/en/latest/contents.html) for unit testing.
+[pytest](https://docs.pytest.org/en/latest/contents.html) library for unit testing.
 
 Assuming you've cloned this repo locally, you can run
 tests by using Pipenv to install dependencies and
@@ -267,3 +268,86 @@ pipenv run pytest
 pipenv shell
 pytest
 ```
+
+### Slow tests
+
+Slow-running tests should be [marked][] as such:
+
+
+```
+@pytest.mark.slow
+def test_something_slow():
+    ...
+
+```
+
+Slow tests are skipped by default. To run them, pass the `--runslow` flag
+when invoking pytest:
+
+```
+pytest --runslow
+```
+
+### Live tests
+
+Tests that hit live web sites should be [marked][] as `webtest`,
+allowing them to be [executed selectively][]:
+
+```
+@pytest.mark.webtest
+def test_that_hits_live_website():
+    ...
+
+# On the command line, run only tests marked as "webtest"
+pytest -m webtest
+```
+
+In many cases, tests that hit live websites should be marked
+as both `webtest` and `slow`:
+
+```
+@pytest.mark.webtest
+@pytest.mark.slow
+def test_that_hits_live_website():
+    ...
+
+# On the command line, you'll need two flags
+pytest --runslow -m webtest
+```
+
+Live web tests of Selenium-based scrapers will open a
+web browser by default. All tests of Selenium scrapers
+should use the `headless` fixture in order to provide
+the ability to disable running tests in browser.
+
+> These tests should typically be marked as `slow` and `webtest`
+> as well.
+
+```
+@pytest.mark.webtest
+@pytest.mark.slow
+def test_selenium_scrape(headless):
+```
+
+You can then activate headless mode when running
+pytest by using the `--headless` flag:
+
+
+```
+pytest --headless --runslow
+```
+
+
+### Test login credentials
+
+Tests that hit [live web sites](#live-tests) may require authentication,
+as in the case of some Odyssey sites such as Dekalb and Chatham counties
+in Georgia.
+
+Such tests require creating user accounts and adding login credentials
+to a local YAML configuration file, as described above in
+the [Configuration section](#configuration).
+
+
+[marked]: https://docs.pytest.org/en/stable/example/markers.html
+[executed selectively]: https://docs.pytest.org/en/stable/example/markers.html#marking-test-functions-and-selecting-them-for-a-run
diff --git a/court_scraper/platforms/odyssey_site/pages/case_detail.py b/court_scraper/platforms/odyssey_site/pages/case_detail.py
index 82e676d..526991e 100644
--- a/court_scraper/platforms/odyssey_site/pages/case_detail.py
+++ b/court_scraper/platforms/odyssey_site/pages/case_detail.py
@@ -7,22 +7,50 @@
 
 class CaseDetailPageLocators:
 
-    FINANCIAL_INFO = (
-        By.CSS_SELECTOR,
-        'div#divFinancialInformation_body h1'
+    SIDENAV_CASE_INFO_LINK = (
+        By.XPATH,
+       '//a[text()="Case Information"]'
     )
 
 class CaseDetailPage(BasePage):
 
-    financials_locator = CaseDetailPageLocators.FINANCIAL_INFO
+    sidenav_case_info_locator = CaseDetailPageLocators.SIDENAV_CASE_INFO_LINK
 
     @property
     def page_source(self):
+        """
+        Page source waits for Case Info to appear
+        in sidebar nav before proceeding.
+
+        This strategy may not be robust enough
+        to handle long-loading page sections.
+
+        See NOTE below for beginnings of an alternative
+        strategy.
+        """
         WebDriverWait(self.driver, 10).until(
-            EC.text_to_be_present_in_element(
-                self.financials_locator,
-                'Financial'
-            )
+            EC.presence_of_element_located(self.sidenav_case_info_locator)
         )
+        """
+        # NOTE: Below are some initial thoughts on an alternate
+        strategy to wait for page load, if the more naive approach
+        above doesn't work consistently for pages with longer-loading
+        h1 sections.
+
+        # Grab the section heads from sidebar nav
+        side_nav = self.driver.find_element(*self.sidenav_locator)
+        section_heads = [
+            a.get_attribute('text') for a in
+            side_nav.find_elements_by_tag_name('a')
+        ]
+        # TODO: Feed section_heads into an XPATH query that
+        uses presence_of_all_elements? Or if XPATH/presence method
+        don't support this, may need more a custom wait
+        condition (https://selenium-python.readthedocs.io/waits.html)
+        or some other approach
+
+        # Example of finding h1 element that could be long-loading
+        self.driver.find_elements_by_xpath('//h1[text()="Documents"]')
+        """
         return self.driver.page_source
 
diff --git a/court_scraper/platforms/odyssey_site/pages/search_results.py b/court_scraper/platforms/odyssey_site/pages/search_results.py
index 999374c..b2d8c6a 100644
--- a/court_scraper/platforms/odyssey_site/pages/search_results.py
+++ b/court_scraper/platforms/odyssey_site/pages/search_results.py
@@ -51,9 +51,7 @@ def __get__(self, obj, owner):
     def _prep_case_rows(self, elements):
         case_rows = []
         for el in elements:
-            row = ResultRow(el)
-            if row.is_case_row:
-                case_rows.append(row)
+            case_rows.append(ResultRow(el))
         return case_rows
 
 
@@ -79,14 +77,6 @@ def metadata(self):
             'case_detail_url': case_detail_url,
         }
 
-    @property
-    def is_case_row(self):
-        case_num = self.inner_text.split('\t')[0]
-        case_num_pattern = r'^\d\d[A-Z0-9]+$'
-        if re.match(case_num_pattern, case_num):
-            return True
-        return False
-
     @property
     def inner_text(self):
         return self.el.get_attribute('innerText').strip('\t')


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index e217149..9c9c6eb 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,7 +1,54 @@
+from os.path import expanduser
 import shutil
 from pathlib import Path
 
 import pytest
+import yaml
+try:
+    from yaml import CLoader as Loader
+except ImportError:
+    from yaml import Loader
+
+
+def pytest_addoption(parser):
+    parser.addoption(
+        "--runslow", action="store_true", default=False, help="run slow tests"
+    )
+    parser.addoption(
+        "--headless", action="store_true", default=False,
+        help="Run live webtests in headless mode"
+    )
+
+
+def pytest_configure(config):
+    config.addinivalue_line("markers", "slow: mark test as slow to run")
+    config.addinivalue_line("markers", "webtest: mark test as hitting live websites")
+
+
+def pytest_collection_modifyitems(config, items):
+    if config.getoption("--runslow"):
+        # --runslow given in cli: do not skip slow tests
+        return
+    skip_slow = pytest.mark.skip(reason="need --runslow option to run")
+    for item in items:
+        if "slow" in item.keywords:
+            item.add_marker(skip_slow)
+
+
+@pytest.fixture
+def headless(request):
+    return request.config.getoption("--headless")
+
+
+@pytest.fixture
+def live_configs():
+    try:
+        home = expanduser("~")
+        config_path = Path(home, '.court-scraper/config.yaml')
+    except KeyError:
+        return ''
+    with open(config_path,'r') as fh:
+        return yaml.load(fh, Loader=Loader)
 
 
 @pytest.fixture
diff --git a/tests/platforms/test_odyssey_site.py b/tests/platforms/test_odyssey_site.py
index b2d8f2c..50a28e7 100644
--- a/tests/platforms/test_odyssey_site.py
+++ b/tests/platforms/test_odyssey_site.py
@@ -26,3 +26,28 @@ def test_login(webdriver_mock, login_page_mock):
     for expected in expected_calls:
         assert expected in actual_calls
 
+
+@pytest.mark.parametrize(
+    "test_input",
+    [
+        (
+            "https://cmsportal.chathamcounty.org/Portal/Home/Dashboard/29",
+            ["MGCV20-00001"]
+        ),
+        (
+            "https://ody.dekalbcountyga.gov/portal/Home/Dashboard/29",
+            ["19d89169"]
+        ),
+    ]
+)
+@pytest.mark.slow
+@pytest.mark.webtest
+def test_search(test_input, headless, live_configs, court_scraper_dir):
+    auth = live_configs['ga_dekalb']
+    username = auth['username']
+    password = auth['password']
+    url, case_ids = test_input
+    site = OdysseySite(url, username, password, download_dir=court_scraper_dir)
+    site.login(headless=headless)
+    results = site.search(search_terms=case_ids, headless=False)
+    assert len(results) == 1