Revert "Add Python 2.4 to Travis config"

This reverts commit 7b0843350b6487ebad2bc9fe62eefc3b231a3bf5.