Yes, you need to use coffee grounds in moderation. They are very acidic and should be reserved only for acid-loving plants like such as hydrangeas and lilies. If your soil already has high nitrogen content, the extra boost from coffee grounds could lead to the stunted growth of fruits and flowers.