The main downsides include lower profit margins due to higher product costs and less control over quality and fulfillment speed compared to managing your own inventory. Additionally, branding options might be limited compared to producing your own products.