snoyberg/conduit — pull request #105 by rycee (2013-07-12 10:19:44+00:00)
Title: [conduit] Optimization of UTF-8 decode

In todays profiling adventures I noticed that `Data.Conduit.Text.decode` is quite slow when decoding UTF-8. Looking a bit closer it seems that it currently uses a somewhat inefficient algorithm for splitting the incoming byte string after the last valid code point. I tried changing the algorithm to take advantage of self-synchonization and got a nice speedup. Using the criterion benchmark

```Haskell
scan' :: L.ByteString -> Either SomeException ()
scan' lbs = C.runException $ CB.sourceLbs lbs $$ CT.decode CT.utf8' =$= CL.sinkNull

scan :: L.ByteString -> Either SomeException ()
scan lbs = C.runException $ CB.sourceLbs lbs $$ CT.decode CT.utf8 =$= CL.sinkNull

main :: IO ()
main =
do
lbs <- runResourceT $ CB.sourceFile "input.txt" $$ CB.sinkLbs
defaultMain [ bench "original" $ whnf scan' lbs
, bench "optimized" $ whnf scan lbs
]
```
where `input.txt` contains 10 copies of a complicated [UTF-8 sample file](http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt) gives results as follows ([html report](http://tau.rycee.net/conduit-be478e6-benchmark.html)):
```
benchmarking original
mean: 971.8244 us, lb 970.1733 us, ub 973.9165 us, ci 0.950
std dev: 9.444848 us, lb 7.836706 us, ub 12.66788 us, ci 0.950

benchmarking optimized
mean: 483.5697 us, lb 482.5582 us, ub 484.8423 us, ci 0.950
std dev: 5.810981 us, lb 4.797644 us, ub 7.817784 us, ci 0.950
```