Repository: sagemath/sage

Issue:
Permutation(n) doesn't use the cache if there is one.
### Is there an existing issue for this?

- [X] I have searched the existing issues for a bug report that matches the one I want to file, without success.

### Did you read the documentation and troubleshoot guide?

- [X] I have read the documentation and troubleshoot guide

### Environment

```markdown
- **OS**: any
- **Sage Version**: 10.0.beta2
```


### Steps To Reproduce

```
sage: %time for x in Permutations(10): pass
CPU times: user 4 s, sys: 0 ns, total: 4 s
Wall time: 4 s
sage: %time for x in Permutations(10).tuple(): pass
CPU times: user 13.2 s, sys: 252 ms, total: 13.5 s
Wall time: 13.5 s
sage: %time for x in Permutations(10).tuple(): pass
CPU times: user 124 ms, sys: 0 ns, total: 124 ms
Wall time: 124 ms
sage: %time for x in Permutations(10): pass
CPU times: user 4.2 s, sys: 0 ns, total: 4.2 s
Wall time: 4.2 s
```

### Expected Behavior

Once the cache of a finite enumerated set is filled, it should be used.

### Actual Behavior

The iterator over permutations doesn't use the cache `self._list`, even if present.

### Additional Information

_No response_


Discussion:
`FiniteEnumeratedSets.ParentMethods._list_from_iterator` contains the following:
```
.. WARNING::

                The overriding of ``self.__iter__`` to use the cache
                is ignored upon calls such as ``for x in C:`` or
                ``list(C)`` (which essentially ruins its purpose).
                Indeed, Python looks up the ``__iter__`` method
                directly in the class of ``C``, bypassing ``C``'s
                dictionary (see the Python reference manual,
                `Special method lookup for new-style classes <http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes>`_)
```
I am guessing that the correct doc is now https://docs.python.org/3/reference/datamodel.html#special-lookup.

If I understand correctly, setting `__iter__` in this method has no effect anymore, right?  So, should `StandardPermutations_n.__iter__` (or, more generally, any class where we implement this caching mechanism) check whether the cache is present?  Or, maybe should we have an `_iter_` method instead, and `__iter__` only be implemented by the category?

@tscrim, what do you think?
I am not sure. The simplest way would be to just test it by adding a `print("hello iterator")` statement.
Sorry, I miscommunicated.  I have tested, that the method is set, but not used, as described in the python reference.  What I do not know whether this might have some weird side effect.  But more importantly, I am not completely sure about the cure.  I see a few possibilities:

1. in every `__iter__` definition of a class in `FiniteEnumeratedSets`, check whether `_list` exists, and if so, `yield from self._list`.
2. remove the feature.
3. Rename `__iter__` to `_iter_` in every class in `FiniteEnumeratedSets`, and make `FiniteEnumeratedSets.ParentMethods.__iter__` either call `_iter_` or `yield from self._list`.  Additionally, I think there could be a `_test_iter` that complains if `__iter__` and not `_iter_` is a method of the class when the `TestSuite` is run.

Additionally, I wonder whether we should and can do something about the following.
```py
sage: P = Permutations(3)
sage: P.__iter__
<bound method StandardPermutations_n.__iter__ of Standard permutations of 3>
sage: tuple(P)
([1, 2, 3], [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 1, 2], [3, 2, 1])
sage: P.__iter__
<bound method StandardPermutations_n.__iter__ of Standard permutations of 3>
sage: P.tuple()
([1, 2, 3], [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 1, 2], [3, 2, 1])
sage: P.__iter__
<bound method EnumeratedSets.ParentMethods._iterator_from_list of Standard permutations of 3>
```
It is not clear to me whether we want the function `tuple` (and `list`) have the same caching effect.  I tend to think that it should, but I also don't know whether it is possible.
Possibly, replacing `__iter__` with `_iter_` is non-trivial, because a category which is not necessarily in `Finite` may implement `__iter__`, and then this definition would be inherited.

Maybe it is best to remove the feature.
It is important to pay attention to what is happening in your examples. `tuple(P)` is essentially doing the same as `tuple(iter(P))`, not calling `P.tuple()`. The iterator has no idea if it actually completes the full iteration or not, nor is it holding onto previously created objects (be design, it should not). There is not a reasonable way to do anything with that example (other than telling the user to use `P.tuple()` instead).

Another way to fix this would be to explicitly change the `__class__` with a new (dynamic?) class with the new `__iter__`. Here is a proof-of-concept:
```
sage: P = Permutations(5)
sage: %time T = tuple(P)
CPU times: user 427 µs, sys: 38 µs, total: 465 µs
Wall time: 506 µs
sage: T = P.tuple()
sage: class PermsNew(P.__class__):
....:     def __iter__(self):
....:         yield from self._list
....:
sage: P.__class__ = PermsNew
sage: %time T = tuple(P)
CPU times: user 44 µs, sys: 0 ns, total: 44 µs
Wall time: 46.3 µs
```
This is the way things are monkey patched when using special methods (albeit somewhat hacky and brittle).
Are you sure that having this caching feature is worthwhile?  It seems that it is a lot of code to support a feature, which is perhaps not really useful.

Maybe it makes sense to present the use case, which is why this was noticed:

For an experiment, Alice wants to check all statistics on permutations in findstat which have code.  A typical example is
```py
def statistic(pi):
    return len([y for y in Permutations(pi.size()) if y.bruhat_lequal(pi)])
```
Thus, if the caching would work as originally intended, one could do
```py
for n in range(8): _ = Permutations(n)
do_experiments_with_permutation_statistics()
```

I am not sure whether this is worth the effort, but apparently, the difference is noticeable.

There are other low hanging fruits to speed up the iterators themselves.  For example:
```py
    def __iter__(self):
        n = 0
        while True:
            for p in itertools.permutations(range(1, n + 1)):
                yield self.element_class(self, p)
            n += 1
```
half the time is spent in checking that the input is actually a permutation:
```py
    def __iter__(self):
        n = 0
        while True:
            for p in itertools.permutations(range(1, n + 1)):
                yield self.element_class(self, p, check_input=False)
            n += 1
```
Possibly we could save even more time by calling `__init__` instead of `__class_call_private__` (which checks whether the input is a string, a pair of tableaux, or a list of cycles.  I am not completely sure how to do that.

Then again, `Permutation` still inherits from `CombinatorialElement`, which is deprecated.
Not everything is as cheap to create as permutations, so I think it would be useful, even more so in tighter loops.

The `element_class` call should be avoiding the `__classcall_private__` because it is a (dynamically created) subclass of `Permutation`. However, big +1 on adding a `check` argument.

There are some subtleties with using `ClonableArray` compared to `CombinatorialElement`, as well as speed differences that are not always faster IIRC. Changing that over deserves its own issue and a lot of care.
It appears that the following *just works*.  I tested `sage/categories`, the only failed tests were those showing that caching is broken.

Shall I prepare a pull request from this?  It does have a side effect apart from the desired one, of course:
```py
sage: P = Permutations(5)
sage: P.__class__
<class 'sage.combinat.permutation.StandardPermutations_n_with_category'>
sage: _ = P.tuple()
sage: P.__class__
<class 'sage.categories.finite_enumerated_sets.FiniteEnumeratedSets.ParentMethods._list_from_iterator.<locals>.ClassWithIter'>
```

```diff
diff --git a/src/sage/categories/finite_enumerated_sets.py b/src/sage/categories/finite_enumerated_sets.py
index 4a872cf2fbf..17f87c53460 100644
--- a/src/sage/categories/finite_enumerated_sets.py
+++ b/src/sage/categories/finite_enumerated_sets.py
@@ -346,10 +346,14 @@ class FiniteEnumeratedSets(CategoryWithAxiom):
             result = tuple(self.__iter__())
             try:
                 self._list = result
-                self.__iter__ = self._iterator_from_list
                 self.cardinality = self._cardinality_from_list
                 self.tuple = self._tuple_from_list
                 self.unrank = self._unrank_from_list
+                class ClassWithIter(self.__class__):
+                    def __iter__(self):
+                        yield from self._list
+                self.__class__ = ClassWithIter
+
             except AttributeError:
                 pass
             return list(result)
```
I think we can fix the output by setting the new class's `__name__` and `__qualname__` to those of `self.__class___` (of course, before setting it). See [this SO post](https://stackoverflow.com/questions/5352781/how-to-set-class-names-dynamically).

Yes, please create a PR.

Resolving patch:
diff --git a/src/sage/combinat/dyck_word.py b/src/sage/combinat/dyck_word.py
--- a/src/sage/combinat/dyck_word.py
+++ b/src/sage/combinat/dyck_word.py
@@ -2194,7 +2194,7 @@ def to_noncrossing_permutation(self) -> Permutation:
                     pi[touches[i] - 1] = pi[touches[i + 1] - 1]
                 pi[touches[-1] - 1] = a
             D, touch_sequence = pealing(D, return_touches=True)
-        return Permutations()(pi, check_input=False)
+        return Permutations()(pi, check=False)
 
     @combinatorial_map(name='to 321 avoiding permutation')
     def to_321_avoiding_permutation(self) -> Permutation:
diff --git a/src/sage/combinat/permutation.py b/src/sage/combinat/permutation.py
--- a/src/sage/combinat/permutation.py
+++ b/src/sage/combinat/permutation.py
@@ -262,6 +262,7 @@
 from sage.libs.gap.libgap import libgap
 from sage.matrix.matrix_space import MatrixSpace
 from sage.misc.cachefunc import cached_method
+from sage.misc.decorators import rename_keyword
 from sage.misc.prandom import sample
 from sage.rings.integer import Integer
 from sage.rings.integer_ring import ZZ
@@ -286,7 +287,7 @@ class Permutation(CombinatorialElement):
 
       - list of integers, viewed as one-line permutation notation. The
         construction checks that you give an acceptable entry. To avoid
-        the check, use the ``check_input`` option.
+        the check, use the ``check`` option.
 
       - string, expressing the permutation in cycle notation.
 
@@ -299,7 +300,7 @@ class Permutation(CombinatorialElement):
         the permutation obtained from the pair using the inverse of the
         Robinson-Schensted algorithm.
 
-    - ``check_input`` (boolean) -- whether to check that input is correct. Slows
+    - ``check`` (boolean) -- whether to check that input is correct. Slows
        the function down, but ensures that nothing bad happens. This is set to
        ``True`` by default.
 
@@ -308,7 +309,7 @@ class Permutation(CombinatorialElement):
         Since :trac:`13742` the input is checked for correctness : it is not
         accepted unless it actually is a permutation on `\{1, \ldots, n\}`. It
         means that some :meth:`Permutation` objects cannot be created anymore
-        without setting ``check_input = False``, as there is no certainty that
+        without setting ``check=False``, as there is no certainty that
         its functions can handle them, and this should be fixed in a much
         better way ASAP (the functions should be rewritten to handle those
         cases, and new tests be added).
@@ -425,7 +426,8 @@ class Permutation(CombinatorialElement):
         []
     """
     @staticmethod
-    def __classcall_private__(cls, l, check_input = True):
+    @rename_keyword(deprecation=35233, check_input='check')
+    def __classcall_private__(cls, l, check=True):
         """
         Return a permutation in the general permutations parent.
 
@@ -480,9 +482,10 @@ def __classcall_private__(cls, l, check_input = True):
                 raise ValueError("cannot convert l (= %s) to a Permutation"%l)
 
         # otherwise, it gets processed by CombinatorialElement's __init__.
-        return Permutations()(l, check_input=check_input)
+        return Permutations()(l, check=check)
 
-    def __init__(self, parent, l, check_input=True):
+    @rename_keyword(deprecation=35233, check_input='check')
+    def __init__(self, parent, l, check=True):
         """
         Constructor. Checks that INPUT is not a mess, and calls
         :class:`CombinatorialElement`. It should not, because
@@ -492,7 +495,7 @@ def __init__(self, parent, l, check_input=True):
 
         - ``l`` -- a list of ``int`` variables
 
-        - ``check_input`` (boolean) -- whether to check that input is
+        - ``check`` (boolean) -- whether to check that input is
           correct. Slows the function down, but ensures that nothing bad
           happens.
 
@@ -519,7 +522,7 @@ def __init__(self, parent, l, check_input=True):
         """
         l = list(l)
 
-        if check_input and len(l) > 0:
+        if check and len(l) > 0:
             # Make a copy to sort later
             lst = list(l)
 
@@ -5738,7 +5741,7 @@ def __iter__(self) -> Iterator[Permutation]:
             []
         """
         for x in itertools.permutations(range(1,self.n+1), int(self._k)):
-            yield self.element_class(self, x)
+            yield self.element_class(self, x, check=False)
 
     def cardinality(self) -> Integer:
         """
@@ -6265,7 +6268,7 @@ def __iter__(self) -> Iterator:
              ['t', 'a', 'c']]
         """
         for p in itertools.permutations(self._set, len(self._set)):
-            yield self.element_class(self, p)
+            yield self.element_class(self, p, check=False)
 
     def cardinality(self) -> Integer:
         """
@@ -6378,7 +6381,8 @@ def __iter__(self):
         mset_list = [lmset.index(x) for x in lmset]
         indices = libgap.Arrangements(mset_list, self._k).sage()
         for ktuple in indices:
-            yield self.element_class(self, [lmset[x] for x in ktuple])
+            yield self.element_class(self, [lmset[x] for x in ktuple],
+                                     check=False)
 
 
 class Permutations_setk(Permutations_set):
@@ -6450,7 +6454,7 @@ def __iter__(self):
             [[1, 2], [1, 4], [2, 1], [2, 4], [4, 1], [4, 2]]
         """
         for perm in itertools.permutations(self._set, int(self._k)):
-            yield self.element_class(self, perm)
+            yield self.element_class(self, perm, check=False)
 
     def random_element(self):
         """
@@ -6646,7 +6650,7 @@ def __iter__(self):
         n = 0
         while True:
             for p in itertools.permutations(range(1, n + 1)):
-                yield self.element_class(self, p)
+                yield self.element_class(self, p, check=False)
             n += 1
 
     def graded_component(self, n):
@@ -6692,7 +6696,8 @@ def __init__(self, n, category=None):
             category = FiniteEnumeratedSets()
         Permutations.__init__(self, category=category)
 
-    def _element_constructor_(self, x, check_input=True):
+    @rename_keyword(deprecation=35233, check_input='check')
+    def _element_constructor_(self, x, check=True):
         """
         Construct an element of ``self`` from ``x``.
 
@@ -6721,7 +6726,7 @@ def _element_constructor_(self, x, check_input=True):
         """
         if len(x) < self.n:
             x = list(x) + list(range(len(x) + 1, self.n + 1))
-        return self.element_class(self, x, check_input=check_input)
+        return self.element_class(self, x, check=check)
 
     def __contains__(self, x):
         """
@@ -6785,7 +6790,7 @@ def __iter__(self):
             [[1, 2, 3], [1, 3, 2], [2, 1, 3], [2, 3, 1], [3, 1, 2], [3, 2, 1]]
         """
         for p in itertools.permutations(range(1, self.n + 1)):
-            yield self.element_class(self, p)
+            yield self.element_class(self, p, check=False)
 
     def _coerce_map_from_(self, G):
         """
@@ -6893,7 +6898,7 @@ def identity(self):
             sage: Permutations(0).identity()
             []
         """
-        return self.element_class(self, range(1,self.n+1))
+        return self.element_class(self, range(1,self.n+1), check=False)
 
     one = identity
 
@@ -6951,7 +6956,8 @@ def random_element(self):
             sage: s in Permutations(4)
             True
         """
-        return self.element_class(self, sample(range(1,self.n+1), self.n))
+        return self.element_class(self, sample(range(1, self.n+1), self.n),
+                                  check=False)
 
     def cardinality(self):
         """
@@ -7041,7 +7047,7 @@ def element_in_conjugacy_classes(self, nu):
             i += nui
         for i in range(nu.size(), self.n):
             l.append(i+1)
-        return self.element_class(self, l)
+        return self.element_class(self, l, check=False)
 
     def conjugacy_classes_representatives(self):
         r"""
@@ -7205,7 +7211,7 @@ def simple_reflection(self, i):
         g = list(range(1, self.n+1))
         g[i-1] = i+1
         g[i] = i
-        return self.element_class(self, g)
+        return self.element_class(self, g, check=False)
 
     class Element(Permutation):
         def has_left_descent(self, i, mult=None):
@@ -7608,7 +7614,7 @@ def from_cycles(n, cycles, parent=None):
     for i in range(n):
         if p[i] is None:
             p[i] = ZZ(i + 1)
-    return parent(p, check_input=False)
+    return parent(p, check=False)
 
 
 def from_lehmer_code(lehmer, parent=None):
@@ -8175,7 +8181,7 @@ def __iter__(self):
             pos += 1
 
         for le in dag.topological_sort_generator():
-            yield self.element_class(self, le)
+            yield self.element_class(self, le, check=False)
 
 
 class StandardPermutations_recoilsfatter(Permutations):
@@ -8247,7 +8253,7 @@ def __iter__(self):
             dag.add_edge(pos+1, pos)
 
         for le in dag.topological_sort_generator():
-            yield self.element_class(self, le)
+            yield self.element_class(self, le, check=False)
 
 
 class StandardPermutations_recoils(Permutations):
@@ -8319,7 +8325,7 @@ def __iter__(self):
             dag.add_edge(pos+1, pos)
 
         for le in dag.topological_sort_generator():
-            yield self.element_class(self, le)
+            yield self.element_class(self, le, check=False)
 
 
 def from_major_code(mc, final_descent=False):
@@ -8382,7 +8388,7 @@ def from_major_code(mc, final_descent=False):
         #Lemma 2.2 in Skandera
 
         #Get the descents of w and place them in reverse order
-        d = Permutation(w, check_input=False).descents(final_descent=final_descent)
+        d = Permutation(w, check=False).descents(final_descent=final_descent)
         d.reverse()
 
         #a is the list of all positions which are not descents
@@ -8394,7 +8400,7 @@ def from_major_code(mc, final_descent=False):
         indices = d + a
         w.insert(indices[l], i)
 
-    return Permutation(w, check_input = False)
+    return Permutation(w, check=False)
 
 ################
 # Bruhat Order #
@@ -8900,11 +8906,11 @@ def __iter__(self, distinct=False):
         """
         if len(self.partition) == 1:
             for i in CyclicPermutations(self.partition[0]).iterator(distinct=distinct):
-                yield self.element_class(self, [i])
+                yield self.element_class(self, [i], check=False)
         else:
             for right in CyclicPermutationsOfPartition(self.partition[1:]).iterator(distinct=distinct):
                 for perm in CyclicPermutations(self.partition[0]).iterator(distinct=distinct):
-                    yield self.element_class(self, [perm] + list(right))
+                    yield self.element_class(self, [perm] + list(right), check=False)
 
     iterator = __iter__
 
@@ -9016,7 +9022,7 @@ def __iter__(self):
         n = 0
         while True:
             for x in itertools.permutations(range(1, n + 1)):
-                x = self.element_class(self, x)
+                x = self.element_class(self, x, check=False)
                 if all(x.avoids(p) for p in self._a):
                     yield x
             n += 1
@@ -9121,7 +9127,7 @@ def __iter__(self):
         """
         if self.n > 0:
             return iter(PatternAvoider(self, self._a))
-        return iter([self.element_class(self, [])])
+        return iter([self.element_class(self, [], check=False)])
 
     def cardinality(self):
         """
@@ -9154,7 +9160,7 @@ def __iter__(self):
             sage: Permutations(3, avoiding=[1,2]).list()
             [[3, 2, 1]]
         """
-        yield self.element_class(self, range(self.n, 0, -1))
+        yield self.element_class(self, range(self.n, 0, -1), check=False)
 
     def cardinality(self):
         """
@@ -9186,7 +9192,7 @@ def __iter__(self):
             sage: Permutations(3, avoiding=[2,1]).list()
             [[1, 2, 3]]
         """
-        yield self.element_class(self, range(1, self.n+1))
+        yield self.element_class(self, range(1, self.n+1), check=False)
 
     def cardinality(self):
         """
@@ -9249,28 +9255,33 @@ def __iter__(self):
 
         elif self.n < 3:
             for p in itertools.permutations(range(1, self.n + 1)):
-                yield self.element_class(self, p)
+                yield self.element_class(self, p, check=False)
             return
 
         elif self.n == 3:
             for p in itertools.permutations(range(1, self.n + 1)):
                 if p != (1, 3, 2):
-                    yield self.element_class(self, p)
+                    yield self.element_class(self, p, check=False)
             return
 
         #Yield all the 132 avoiding permutations to the right.
         for right in StandardPermutations_avoiding_132(self.n - 1):
-            yield self.element_class(self, [self.n] + list(right))
+            yield self.element_class(self, [self.n] + list(right),
+                                     check=False)
 
         #yi
         for i in range(1, self.n-1):
             for left in StandardPermutations_avoiding_132(i):
                 for right in StandardPermutations_avoiding_132(self.n-i-1):
-                    yield self.element_class(self, [x+(self.n-i-1) for x in left] + [self.n] + list(right) )
+                    yield self.element_class(self,
+                                             [x + (self.n-i-1) for x in left]
+                                             + [self.n] + list(right),
+                                             check=False)
 
         #Yield all the 132 avoiding permutations to the left
         for left in StandardPermutations_avoiding_132(self.n - 1):
-            yield self.element_class(self, list(left) + [self.n])
+            yield self.element_class(self, list(left) + [self.n],
+                                     check=False)
 
 
 class StandardPermutations_avoiding_123(StandardPermutations_avoiding_generic):
@@ -9310,13 +9321,13 @@ def __iter__(self):
 
         elif self.n < 3:
             for p in itertools.permutations(range(1, self.n + 1)):
-                yield self.element_class(self, p)
+                yield self.element_class(self, p, check=False)
             return
 
         elif self.n == 3:
             for p in itertools.permutations(range(1, self.n + 1)):
                 if p != (1, 2, 3):
-                    yield self.element_class(self, p)
+                    yield self.element_class(self, p, check=False)
             return
 
         for p in StandardPermutations_avoiding_132(self.n):
@@ -9342,7 +9353,7 @@ def __iter__(self):
                     new_p.append( non_minima[b] )
                     b += 1
 
-            yield self.element_class(self, new_p)
+            yield self.element_class(self, new_p, check=False)
 
 
 class StandardPermutations_avoiding_321(StandardPermutations_avoiding_generic):
@@ -9374,7 +9385,7 @@ def __iter__(self):
             [[2, 3, 1], [3, 1, 2], [1, 3, 2], [2, 1, 3], [1, 2, 3]]
         """
         for p in StandardPermutations_avoiding_123(self.n):
-            yield self.element_class(self, p.reverse())
+            yield self.element_class(self, p.reverse(), check=False)
 
 
 class StandardPermutations_avoiding_231(StandardPermutations_avoiding_generic):
@@ -9406,7 +9417,7 @@ def __iter__(self):
             [[3, 2, 1], [3, 1, 2], [1, 3, 2], [2, 1, 3], [1, 2, 3]]
         """
         for p in StandardPermutations_avoiding_132(self.n):
-            yield self.element_class(self, p.reverse())
+            yield self.element_class(self, p.reverse(), check=False)
 
 
 class StandardPermutations_avoiding_312(StandardPermutations_avoiding_generic):
@@ -9438,7 +9449,7 @@ def __iter__(self):
             [[3, 2, 1], [2, 3, 1], [2, 1, 3], [1, 3, 2], [1, 2, 3]]
         """
         for p in StandardPermutations_avoiding_132(self.n):
-            yield self.element_class(self, p.complement())
+            yield self.element_class(self, p.complement(), check=False)
 
 
 class StandardPermutations_avoiding_213(StandardPermutations_avoiding_generic):
@@ -9508,8 +9519,10 @@ def _rec(self, obj, state):
             yld = True
 
         for pos in reversed(range(len(obj)+1)):
-            new_obj = self._parent.element_class(self._parent, obj[:pos] + [i] + obj[pos:])
-            if all( not new_obj.has_pattern(p) for p in self._patterns):
+            new_obj = self._parent.element_class(self._parent,
+                                                 obj[:pos] + [i] + obj[pos:],
+                                                 check=False)
+            if all(not new_obj.has_pattern(p) for p in self._patterns):
                 yield new_obj, new_state, yld
 
 


Test patch: