Yes, you can. Reykjavik itself has a very compact, walkable city center area.