Changed the .gitignore file for ignoring the build directory. Only need to ignore the contents and not the folder. Otherwise the packer build will fail.