johnmyleswhite/TextAnalysis.jl — pull request #5 by tanmaykm (2013-09-16 11:06:02+00:00)
Title: optimizing preprocessing by combined regex

Gives around 3x performance boost in a small test. Test involved creating simple inverted indices for text files (https://github.com/tanmaykm/julia_examples/blob/master/dinvidx/dinvidx2.jl)

Before:
````
julia> require("../dinvidx2.jl")

julia> @time create_index()
created 3 part indices
elapsed time: 30.271816734 seconds (3732874940 bytes allocated)

julia> @time create_index()
created 3 part indices
elapsed time: 27.022503601 seconds (3644058912 bytes allocated)
````

After:
````
julia> require("../dinvidx2.jl")

julia> @time create_index()
created 3 part indices
elapsed time: 11.244305093 seconds (849928456 bytes allocated)

julia> @time create_index()
created 3 part indices
elapsed time: 7.954268358 seconds (755496084 bytes allocated)
````