Yes, returning a gift is surely not acceptable manners. Instead, you can find more polite ways to let the guest know by politely asking them if you can exchange it for something better or just by saying thank you for the time and passing it to someone else or gift it to someone whom you think would love the gift.