Only add source after dependency download

This lets docker cache the ADD instruction, giving us a faster build.