Victoria's Inner harbor is much closer to whale-watching grounds than what you can see in Vancouver. The best times for whale watching are when the waters are warmer like in late April to early October.