Yes you can! If you choose to make this an iced latte, I would use only 1/2 cup of water just because the ice will melt and add water to the drink. This is a small amount of water so make sure it covers the tea bag. We don't want it to evaporate and burn.