I had a lovely stay at Jag’s AirBnB. While it is not directly in downtown, it is just a quick bus ride away. I actually preferred the calm and quiet atmosphere of the neighbourhood. Do bare in mind the bus schedules on weekend mornings and nights as they do not run that often in Victoria!