It does depend on the types of activities you want to do. I'd recommend that you spend at least three nights in Zermatt to have time to do multiple activities and not rush through things.