dimroc/etl-language-comparison — pull request #2 by josevalim (2014-10-05 15:02:21+00:00)
Title: Update to Elixir 1.0

Hello @dimroc!

I took the liberty of upgrading your project to Elixir 1.0.

Meanwhile I have noticed two things:

1. You were using a Map on the reduce stage and currently maps can only handle efficiently a small set of keys. Most of your performance slow down came from this and I changed it to a HashDict. We plan to improve Maps so they can handle small and large sets of keys efficiently so this is not a common pitfall in the future

2. The second commit is protocol consolidation. Because of how the VM loads modules, protocol dispatches are expensive during development because we need to check if a new version of the module is available. For production, the recommended approach is to consolidate protocols. So I have updated the script to do so.

Those were very low hanging fruit optimizations and I was able to make the time down to 1m50s on an old macbook pro with 4 cores. If you plan to rerun your tests after this pull request, I would love to know your new results!