Revert "Fix travis build by adding m4 dir"

This reverts commit 5d9a7ec36c7dded1ca21a1ff7472c299ceaf0d7e.