Only build the required files into the gem