Some of the downsides of buying physical gold include huge storage costs, risk of theft or loss, high making charges on gold jewellery, no form of regular or stable returns from buying physical gold, etc.