This depends on your interests, really. If you're a homebody and just wish to relax, unwind, and read a book then a couple of days is good. If you wish to explore the town, harbour, and nearby attractions then I'd say anywhere between 2 and 5 days is a good amount of time to discover Poole and surrounds.