audreyt/regex-pcre-builtin — pull request #5 by mcmtroffaes (2014-04-27 10:11:23+00:00)
Title: Enable UTF8 as a default regex compile option.

This fixes UTF8 regular expressions at least for bytestrings; it remains broken for strings. (One might argue that compUTF8 should only be enabled for strings and not for bytestrings; there is probably a way of doing this but I could not figure it out from the code so far... This patch enables utf8 for both strings and bytestrings.)

Test code:

```haskell
import qualified Data.ByteString.UTF8 as B
import Text.Regex.PCRE

tests = [
("ℝ X", "[^ ]"),
("ℝ X", "[^ ]+")
]

testbytestring :: (String, String) -> IO ()
testbytestring (txt, pat) = do
putStrLn $ "bytestring test: " ++ show btxt ++ " ~= " ++ show bpat
putStrLn $ show (btxt =~ bpat :: B.ByteString)
where btxt = B.fromString txt
bpat = B.fromString pat

teststring :: (String, String) -> IO ()
teststring (txt, pat) = do
putStrLn $ "string test: " ++ txt ++ " ~= " ++ pat
putStrLn $ show $ B.fromString (txt =~ pat :: String)

main :: IO ()
main = do
putStrLn $ "configUTF8 is " ++ show configUTF8
mapM_ teststring tests
mapM_ testbytestring tests
```

which now gives

```
configUTF8 is True
string test: ℝ X ~= [^ ]
ℝ X
string test: ℝ X ~= [^ ]+
ℝ X
bytestring test: "\226\132\157 X" ~= "[^ ]"
"\226\132\157"
bytestring test: "\226\132\157 X" ~= "[^ ]+"
"\226\132\157"
```

Before this patch, the result is:

```
configUTF8 is True
string test: ℝ X ~= [^ ]
ℝ
string test: ℝ X ~= [^ ]+
ℝ X
bytestring test: "\226\132\157 X" ~= "[^ ]"
"\226"
bytestring test: "\226\132\157 X" ~= "[^ ]+"
"\226\132\157"
```

(Note the odd inconsistency between the two string tests in the latter case...)