No, when you have a self-directed IRA with physical gold and silver, you cannot hold the metals yourself. The metals must be stored in a secure facility, typically an IRS-approved depository. This ensures the compliance of your gold and silver holdings with IRS regulations.