It depends on how long you want to stay but, ideally, you want to spend a minimum of seven days in Oaxaca.  This gives you ample time to explore the main areas at a reasonable pace instead of rushing around trying to see everything at once. It also gives you extra time to immerse yourself into the culture and environment.