Yes, you can. The tap water in Iceland is safe to drink. Don’t forget to carry a reusable water bottle for that!