Yes, very much so. Tap water in Germany is even considered a food product and goes through regular quality checks as such. So if you take a reusable water bottle when packing for your short trip to Germany, feel free to refill it in your hotel to save some money (and help the environment).