The main downsides of an FHA loan include limitations to primary residences, stricter appraisal guidelines, and potential challenges with loan assumability. Additionally, there may be restrictions on investment properties.