nebius/SWE-bench-extra huggingface dataset:
Repository: dbcli/litecli

Issue:
TabularOutputFormatter chokes on values that can't be converted to UTF-8 if it's a text column
I don't know how to solve this.  If there is a single record with a non-convertible column it won't print anything.

`Could not decode to UTF-8 column 'verifier' with text '`'���U'`
sqlite3 prints
```
2bho15FMrSQhKAYnjBqRQ1x4LS3zcHANsRjKMJyiSwT9|GnyZktv2SaCehfNCGjoYdAgAirxpCjvBCUXH6MiEHEH7                                                                                        
`'ŜU|`'ŜU 
```

Seeing this was actually helpful because it notified me that I had garbage data, but I still would've thought the other rows would print.

Comments:

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b8c8122..0ccb53a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,7 +1,5 @@
 ## Unreleased - TBD
 
-<!-- add upcoming change notes here -->
-
 ### Features
 
 - Add verbose feature to `favorite_query` command. (Thanks: [Zhaolong Zhu])
@@ -12,6 +10,7 @@
 ### Bug Fixes
 
 - Fix compatibility with sqlparse >= 0.4.0. (Thanks: [chocolateboy])
+- Fix invalid utf-8 exception. (Thanks: [Amjith])
 
 ## 1.4.1 - 2020-07-27
 
diff --git a/litecli/sqlexecute.py b/litecli/sqlexecute.py
index 7ef103c..93acd91 100644
--- a/litecli/sqlexecute.py
+++ b/litecli/sqlexecute.py
@@ -17,6 +17,13 @@ _logger = logging.getLogger(__name__)
 # })
 
 
+def utf8_resilient_decoder(s):
+    try:
+        return s.decode("utf-8")
+    except UnicodeDecodeError:
+        return s.decode("latin-1")
+
+
 class SQLExecute(object):
 
     databases_query = """
@@ -61,6 +68,7 @@ class SQLExecute(object):
             raise Exception("Path does not exist: {}".format(db_dir_name))
 
         conn = sqlite3.connect(database=db_name, isolation_level=None)
+        conn.text_factory = utf8_resilient_decoder
         if self.conn:
             self.conn.close()
 


Test patch:
diff --git a/tests/test_sqlexecute.py b/tests/test_sqlexecute.py
index 03e7f19..0ddfb8f 100644
--- a/tests/test_sqlexecute.py
+++ b/tests/test_sqlexecute.py
@@ -101,6 +101,17 @@ def test_unicode_support_in_output(executor):
     assert_result_equal(results, headers=["t"], rows=[(u"é",)])
 
 
+@dbtest
+def test_invalid_unicode_values_dont_choke(executor):
+    run(executor, "create table unicodechars(t text)")
+    # \xc3 is not a valid utf-8 char. But we can insert it into the database
+    # which can break querying if not handled correctly.
+    run(executor, u"insert into unicodechars (t) values (cast(x'c3' as text))")
+
+    results = run(executor, u"select * from unicodechars")
+    assert_result_equal(results, headers=["t"], rows=[(u"Ã",)])
+
+
 @dbtest
 def test_multiple_queries_same_line(executor):
     results = run(executor, "select 'foo'; select 'bar'")
@@ -199,11 +210,7 @@ def test_verbose_feature_of_favorite_query(executor):
 
     results = run(executor, "\\f sh_param 1")
     assert_result_equal(
-        results,
-        title=None,
-        headers=["a", "id"],
-        rows=[("abc", 1)],
-        auto_status=False,
+        results, title=None, headers=["a", "id"], rows=[("abc", 1)], auto_status=False,
     )
 
     results = run(executor, "\\f+ sh_param 1")
Repository: deardurham/ciprs-reader

Issue:
Parsers do not catch numbered offense rows
The offense row parser is designed to capture data from offense rows that look like this:

ACTION DESCRIPTION SEVERITY LAW CODE

However, on some records they look like this:

\# ACTION DESCRIPTION SEVERITY LAW

With # being either a zero padded number or blank

Comments:

Patch:
diff --git a/ciprs/parsers.py b/ciprs/parsers.py
index 9f2bfec..c648277 100644
--- a/ciprs/parsers.py
+++ b/ciprs/parsers.py
@@ -113,6 +113,25 @@ class OffenseRecordRow(Parser):
         report["Offense Record"]["Records"].append(record)
 
 
+class OffenseRecordRowWithNumber(Parser):
+    """
+    Extract offense row like:
+        54  CHARGED  SPEEDING  INFRACTION  G.S. 20-141(B)
+    """
+
+    # pylint: disable=line-too-long
+    pattern = r"(?:\d*)?\s*(?P<action>\w+)\s+(?P<desc>[\w \-\(\)]+)[ ]{2,}(?P<severity>\w+)[ ]{2,}(?P<law>[\w. \-\(\)]+)"
+
+    def extract(self, matches, report):
+        record = {
+            "Action": matches["action"],
+            "Description": matches["desc"],
+            "Severity": matches["severity"],
+            "Law": matches["law"],
+        }
+        report["Offense Record"]["Records"].append(record)
+
+
 class OffenseDisposedDate(Parser):
 
     pattern = r".*Disposed on:\s*(?P<value>[\d/:]+)"
diff --git a/ciprs/reader.py b/ciprs/reader.py
index b084926..81f9faa 100644
--- a/ciprs/reader.py
+++ b/ciprs/reader.py
@@ -6,6 +6,7 @@ from ciprs.parsers import (
     CaseDetails,
     CaseStatus,
     OffenseRecordRow,
+    OffenseRecordRowWithNumber,
     OffenseDateTime,
     OffenseDisposedDate,
     CaseWasServedOnDate,
@@ -35,6 +36,7 @@ class PDFToTextReader:
             CaseDetails(self.report),
             CaseStatus(self.report),
             OffenseRecordRow(self.report),
+            OffenseRecordRowWithNumber(self.report),
             OffenseDateTime(self.report),
             OffenseDisposedDate(self.report),
             CaseWasServedOnDate(self.report),


Test patch:
diff --git a/tests/test_parsers.py b/tests/test_parsers.py
index ca14731..efa4bb8 100644
--- a/tests/test_parsers.py
+++ b/tests/test_parsers.py
@@ -39,6 +39,16 @@ def test_offense_record_charged():
     assert matches["code"] == "4450"
 
 
+def test_offense_record_charged_with_number():
+    string = "54  CHARGED       SPEEDING(80 mph in a 65 mph zone)    INFRACTION    G.S. 20-141(B)"  # noqa
+    matches = parsers.OffenseRecordRowWithNumber().match(string)
+    assert matches is not None, "Regex match failed"
+    assert matches["action"] == "CHARGED"
+    assert matches["desc"] == "SPEEDING(80 mph in a 65 mph zone)"
+    assert matches["severity"] == "INFRACTION"
+    assert matches["law"] == "G.S. 20-141(B)"
+
+
 def test_offense_record_arrainged():
     string = "ARRAIGNED SPEEDING(80 mph in a 65 mph zone)        INFRACTION    G.S. 20-141(B)    4450"  # noqa
     matches = parsers.OffenseRecordRow().match(string)
Repository: deardurham/ciprs-reader

Issue:
County names with spaces are not parsed properly
The [CaseDetails](https://github.com/deardurham/ciprs-reader/blob/master/ciprs_reader/parser/section/header.py#L12) parser doesn't expect county names to contain spaces, so a line like

```
Case Summary for Court Case: NEW HANOVER 20CR000000
```

will result in a county name of ``NEW``.

AC
- [CaseDetails](https://github.com/deardurham/ciprs-reader/blob/master/ciprs_reader/parser/section/header.py#L12) parser updated to expect spaces
- A test is added wi spaces in the county name

Comments:

Patch:
diff --git a/ciprs_reader/parser/section/header.py b/ciprs_reader/parser/section/header.py
index 1f18711..34edc94 100644
--- a/ciprs_reader/parser/section/header.py
+++ b/ciprs_reader/parser/section/header.py
@@ -13,7 +13,7 @@ class CaseDetails(HeaderParser):
     """Extract County and File No from header on top of first page"""
 
     pattern = (
-        r"\s*Case (Details|Summary) for Court Case[\s:]+(?P<county>\w+) (?P<fileno>\w+)"
+        r"\s*Case (Details|Summary) for Court Case[\s:]+(?P<county>(\w\s*)+) (?P<fileno>\w+)"
     )
 
     def extract(self, matches, report):


Test patch:
diff --git a/tests/parsers/test_header.py b/tests/parsers/test_header.py
index 3610ac9..5fb3138 100644
--- a/tests/parsers/test_header.py
+++ b/tests/parsers/test_header.py
@@ -12,6 +12,14 @@ CASE_DETAIL_DATA = [
         {"county": "ORANGE", "fileno": "99FN9999999"},
         " Case Summary for Court Case: ORANGE 99FN9999999",
     ),
+    (
+        {"county": "NEW HANOVER", "fileno": "00GR000000"},
+        "  Case Details for Court Case NEW HANOVER 00GR000000  ",
+    ),
+    (
+        {"county": "OLD HANOVER", "fileno": "99FN9999999"},
+        " Case Summary for Court Case: OLD HANOVER 99FN9999999",
+    ),
 ]