Repository: openstates/openstates-scrapers

Issue:
TN: Only scraping Joint committees
Running the  [TN committee scraper](http://github.com/openstates/openstates/blob/7506cd3d6e0dacf891a23818d012ff0adf789265/openstates/tn/committees.py) locally, only the 4 joint committees are scraped:

```
18:35:44 INFO scrapelib: GET - http://www.capitol.tn.gov/senate/committees/
18:35:44 INFO scrapelib: GET - http://www.capitol.tn.gov/joint/
18:35:45 INFO scrapelib: GET - http://www.capitol.tn.gov/joint/committees/fiscal-review
18:35:46 INFO billy: Save committee Fiscal Review
18:35:46 INFO scrapelib: GET - http://www.capitol.tn.gov/joint/committees/gov-opps
18:35:47 INFO scrapelib: GET - http://www.capitol.tn.gov/senate/committees/gov-opps.html
18:35:48 INFO scrapelib: GET - http://www.capitol.tn.gov/house/committees/gov-opps.html
18:35:49 INFO billy: Save committee Government Operations
18:35:49 INFO scrapelib: GET - http://www.capitol.tn.gov/joint/committees/pension-ins.html
18:35:50 INFO billy: Save committee Pensions and Insurance
18:35:50 INFO scrapelib: GET - http://www.tn.gov/tacir/
18:35:52 INFO billy: Save committee TACIR
18:35:52 INFO scrapelib: GET - http://www.capitol.tn.gov/house/committees/
18:35:52 INFO billy: imported 4 committee files
```

This matches the [bobsled report](http://bobsled.openstates.org/run-tn-2017-03-17.html) and [TN committees available on openstates.org](https://openstates.org/tn/committees/?chamber=)


Discussion:
Going to tackle this, looks like a quick fix to the xpath selector.

Resolving patch:
diff --git a/openstates/tn/committees.py b/openstates/tn/committees.py
--- a/openstates/tn/committees.py
+++ b/openstates/tn/committees.py
@@ -28,6 +28,10 @@ def fix_whitespace(s):
     return re.sub(r'\s+', ' ', s)
 
 
+# All links in a section with a given title
+COMMITTEE_LINKS_TEMPLATE = '//h2[text()="{header}"]/parent::*//a'
+
+
 class TNCommitteeScraper(CommitteeScraper):
     jurisdiction = 'tn'
     base_href = 'http://www.capitol.tn.gov'
@@ -51,8 +55,9 @@ def scrape_senate_committees(self, url):
         page = lxml.html.fromstring(page)
         page.make_links_absolute(url)
 
-        find_expr = 'body/div/div/h1[text()="Senate Committees"]/' \
-                'following-sibling::div/div/div/div//a'
+        standing = COMMITTEE_LINKS_TEMPLATE.format(header="Standing Committees")
+        select = COMMITTEE_LINKS_TEMPLATE.format(header="Select Committees")
+        find_expr = '{}|{}'.format(standing, select)
         links = [(a.text_content(), a.attrib['href']) for a in
                  page.xpath(find_expr)]
 
@@ -64,10 +69,9 @@ def scrape_house_committees(self, url):
         doc = lxml.html.fromstring(html)
         doc.make_links_absolute(url)
 
-        links = doc.xpath(
-                'body/div/div/h1[text()="House Committees"]/'
-                'following-sibling::div/div/div/div//a'
-                )
+        standing = COMMITTEE_LINKS_TEMPLATE.format(header="Committees & Subcommittees")
+        select = COMMITTEE_LINKS_TEMPLATE.format(header="Select Committees")
+        links = doc.xpath('{}|{}'.format(standing, select))
 
         for a in links:
             self._scrape_committee(a.text.strip(), a.get('href'), 'lower')
@@ -123,10 +127,7 @@ def scrape_joint_committees(self):
         page = lxml.html.fromstring(page)
         page.make_links_absolute(main_url)
 
-        for el in page.xpath(
-                '//div/h2[text()="Committees"]/'
-                'following-sibling::div/div//a'
-                ):
+        for el in page.xpath(COMMITTEE_LINKS_TEMPLATE.format(header="Committees")):
             com_name = el.text
             com_link = el.attrib["href"]
             self.scrape_joint_committee(com_name, com_link)


Test patch: