Only remove a sugar source if no ants are waiting at it.