Repository: openstates/openstates-scrapers

Issue:
SC failing since at least 2019-02-07
SC has been failing since 2019-02-07

Based on automated runs it appears that SC has not run successfully in 7 days (2019-02-07).


```
  04:02:36 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=580
04:02:38 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=590
04:02:38 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=600
04:02:40 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=610
04:02:41 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=620
04:02:42 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=625
04:02:43 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=630
04:02:44 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=640
04:02:45 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=680
04:02:46 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=690
04:02:46 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=700
04:02:48 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=705
04:02:48 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=710
04:02:50 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=730
04:02:51 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=750
04:02:52 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=760
04:02:53 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=780
04:02:54 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=785
04:02:54 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=787
04:02:55 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=790
04:02:57 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=800
04:02:58 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=810
04:03:04 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=820
04:03:06 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=830
04:03:06 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=840
04:03:07 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=850
04:03:08 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=860
04:03:09 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=870
04:03:11 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=880
04:03:12 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=890
04:03:14 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=910
04:03:14 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=920
04:03:15 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=930
04:03:16 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=940
04:03:18 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=950
04:03:18 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=970
04:03:19 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=980
04:03:20 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=990
04:03:21 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1000
04:03:23 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1010
04:03:23 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1020
04:03:25 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1040
04:03:26 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1050
04:03:26 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1070
04:03:27 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1080
04:03:28 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1090
04:03:30 INFO scrapelib: GET - https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1120
04:04:00 WARNING scrapelib: got ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) sleeping for 10 seconds before retry
04:04:41 WARNING scrapelib: got ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) sleeping for 20 seconds before retry
04:05:32 WARNING scrapelib: got ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read)) sleeping for 40 seconds before retry
```

Visit http://bobsled.openstates.org for more info.



Discussion:
The URLs 

* https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1120
* https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1130

both hang with a loading indicator. It seems likely that the responses are simply too big for something on their server; they include many lengthy bill summaries.  See e.g. https://www.scstatehouse.gov/subjectsearch.php?AORB=B&session=123&indexcode=1140 for a working one.  All other indexcodes succeed.

 **If someone has an existing contact with the web folks in the SC state house, please use it; I'll ping them Friday if there's no response here.**  

For a plan B, these two `indexcode`s could be skipped, though I fear this _might_ cause event-import to break from missing bills.  I used this method to verify there were no more problem indexcodes.
Their server has a misconfiguration that's causing it to close the connection early; i have a patch for this that I apparently didn't commit; will drop it shortly.

Resolving patch:
diff --git a/openstates/sc/bills.py b/openstates/sc/bills.py
--- a/openstates/sc/bills.py
+++ b/openstates/sc/bills.py
@@ -9,6 +9,7 @@
 from pupa.scrape import Scraper, Bill, VoteEvent
 from pupa.utils.generic import convert_pdf
 import lxml.html
+import urllib
 
 # Workaround to prevent chunking error (thanks @showerst)
 #
@@ -156,7 +157,18 @@ def scrape_subjects(self, session):
             code = option.get('value')
             url = '%s?AORB=B&session=%s&indexcode=%s' % (subject_search_url,
                                                          session_code, code)
-            data = self.get(url).text
+
+            # SC's server is sending some noncomplient server responses
+            # that are confusing self.get
+            # workaround via
+            # https://stackoverflow.com/questions/14442222/how-to-handle-incompleteread-in-python
+            try:
+                self.info(url)
+                data = urllib.request.urlopen(url).read()
+            except (http.client.IncompleteRead) as e:
+                self.warning("Client IncompleteRead error on {}".format(url))
+                data = e.partial
+
             doc = lxml.html.fromstring(data)
             for bill in doc.xpath('//span[@style="font-weight:bold;"]'):
                 match = re.match(r'(?:H|S) \d{4}', bill.text)


Test patch: