Clean up and reorganize all the build optimizations