Repository: dedupeio/dedupe (language: python)

Issue:
Inconsistent alphaNumericPredicate behavior
Hi, I believe `alphaNumericPredicate` implementation is inconsistent.

Example:
```
In [1]: from dedupe.predicates import alpha_numeric

In [2]: alpha_numeric("a_1")
Out[2]: ['a', '1']

In [3]: alpha_numeric("a 1")
Out[3]: ['1']

In [4]: alpha_numeric("a")
Out[4]: []

In [5]: alpha_numeric("1")
Out[5]: ['1']
```

Should it really match `'a'` for `'a_1'` input, if it doesn't match it for `'a 1'` or just `'a'`?


Discussion / hints:
thanks for the bug report! we'll get into it!
What is desired behavior? a set of all alphanumeric tokens, with the boundary between tokens being all non-alphanumeric characters?

> Should it really match 'a' for 'a_1' input, if it doesn't match it for 'a 1' or just 'a'?

@Nephirus I don't understand what you're saying here, can you please elaborate or phrase that differently?

Is this what we're looking for:
```
"a_1" -> a, 1
"a 1" -> a, 1
"a,1" -> a, 1
"a1 b2" -> a1, b2
"a1b2" -> a1b2
"a" -> a
"1" -> 1
```
?
I'm not really sure what is the desired behavior here.

>> Should it really match 'a' for 'a_1' input, if it doesn't match it for 'a 1' or just 'a'?

> @Nephirus I don't understand what you're saying here, can you please elaborate or phrase that differently?

From the [regex](https://github.com/dedupeio/dedupe/blob/main/dedupe/predicates.py#L24) I would expect it to match all alphanumeric tokens (`[a-zA-Z\d]+`) that contain at least one digit. But `\w` in the lookahead part makes it confusing, since underscore isn't considered a token separator in the lookahead, but also isn't captured as a part of the token. So IMO it should either do this:
```
"a_1" -> a, 1
"a 1" -> a, 1
"a" -> a
"1" -> 1
```
or this:
```
"a_1" -> 1
"a 1" -> 1
"a" -> nothing
"1" -> 1
```

I'm thinking the intended behavior is just `(?=[a-zA-Z\d]*\d)[a-zA-Z\d]+`, but I'm not really sure if it's supposed to work only with ascii characters.

Fix patch:
diff --git a/dedupe/predicates.py b/dedupe/predicates.py
index 288e66657..8d76295a4 100644
--- a/dedupe/predicates.py
+++ b/dedupe/predicates.py
@@ -27,7 +27,7 @@
 start_word = re.compile(r"^([\w']+)").match
 two_start_words = re.compile(r"^([\w']+\s+[\w']+)").match
 start_integer = re.compile(r"^(\d+)").match
-alpha_numeric = re.compile(r"(?=\w*\d)[a-zA-Z\d]+").findall
+alpha_numeric = re.compile(r"(?=[a-zA-Z]*\d)[a-zA-Z\d]+").findall
 
 PUNCTABLE = str.maketrans("", "", string.punctuation)
 


Test patch:
diff --git a/tests/test_predicates.py b/tests/test_predicates.py
index 541339d3f..5c3c1c622 100644
--- a/tests/test_predicates.py
+++ b/tests/test_predicates.py
@@ -68,6 +68,24 @@ def test_precise_latlong(self):
         assert block_val == ()
 
 
+class TestAlpaNumeric(unittest.TestCase):
+    def test_alphanumeric(self):
+
+        assert predicates.alphaNumericPredicate("a1") == set(["a1"])
+        assert predicates.alphaNumericPredicate("1a") == set(["1a"])
+        assert predicates.alphaNumericPredicate("a1b") == set(["a1b"])
+        assert predicates.alphaNumericPredicate("1 a") == set(["1"])
+        assert predicates.alphaNumericPredicate("a1 b1") == set(["a1", "b1"])
+        assert predicates.alphaNumericPredicate("asdf") == set()
+        assert predicates.alphaNumericPredicate("1") == set(["1"])
+        assert predicates.alphaNumericPredicate("a_1") == set(["1"])
+        assert predicates.alphaNumericPredicate("a$1") == set(["1"])
+        assert predicates.alphaNumericPredicate("a 1") == set(["1"])
+        assert predicates.alphaNumericPredicate("773-555-1676") == set(
+            ["773", "555", "1676"]
+        )
+
+
 class TestNumericPredicates(unittest.TestCase):
     def test_order_of_magnitude(self):
         assert predicates.orderOfMagnitude(10) == ("1",)