Repository: sagemath/sage

Issue:
make map_coefficients in modules_with_basis safer or more general
<div id="comment:0"></div>

We would love to be able to, for example, present a symmetric function with factored coefficients.  But:

```
sage: f = SymmetricFunctions(ZZ).s()([2,2,1])
sage: f.map_coefficients(factor)
0
```

The result is not against the specification, because `map_coefficients` requires an endofunction.

There are two issues:

1.) we possibly want to be warned when supplying `map_coefficients` with something that is not an endofunction.  This might be achieved by adding an optional argument `check` with default `True`.

2.) we want some way to factor the coefficients.  Nicolas proposed to make something like

```
sage: f.map_coefficients(factor, codomain=SR)
```
or

```
sage: f.change_base_ring(SR).map_coefficients(factor) 
```
work.

Note that the `Polynomial` class contains something similar, namely:

```
def map_coefficients(self, f, new_base_ring=None)
```

However, there might be another tiny problem: it seems that currently it is not possible to make factored integers into a commutative ring.  This should possibly be a separate ticket.

CC:  @zabrocki @darijgr @sagetrac-sage-combinat @hivert

Component: **algebra**

Keywords: **map_coefficients, modules_with_basis, base_ring, check arguments**

Author: **Martin Rubey**

Branch/Commit: **[public/ticket/18264](https://github.com/sagemath/sagetrac-mirror/tree/public/ticket/18264) @ [`d0d80e7`](https://github.com/sagemath/sagetrac-mirror/commit/d0d80e73ccae7a5823e8692e72e0c31e59800e1b)**

Reviewer: **Nicolas M. Thiéry**

_Issue created by migration from https://trac.sagemath.org/ticket/18264_




Discussion:
Branch: **[u/mantepse/make_map_coefficients_in_modules_with_basis_safer_or_more_general](https://github.com/sagemath/sagetrac-mirror/tree/u/mantepse/make_map_coefficients_in_modules_with_basis_safer_or_more_general)**
<div id="comment:3" align="right">comment:3</div>

Doing this I noticed that `map_coefficients` did not make use of the `distinct` feature of `sum_of_terms`.  I added this too, assuming that initially an element has distinct monomials.
Commit: **[`3a7761e`](https://github.com/sagemath/sagetrac-mirror/commit/3a7761e95b8ca8b83e08635828a3754d751fe918)**
Author: **Martin Rubey**
<div id="comment:7" align="right">comment:7</div>

Hi Martin!

Thanks for taking action; this is looking good!

Now that I am reading actual code: a usual idiom to check that a
coefficient is indeed in the base ring is to coerce it back there with
`R(c)`. I guess that would be a fine behavior: it's faster, and it
takes care of all cases where someone passes an int, or something that
requires some small coercion to be put back in the base ring.

Maybe the option shall be named `coerce` after all. Comments anyone?

Note by the way that coercing would fail in Mike's original use case,
so whether we go for `coerce` or `check` don't make a difference
there:

```
sage: ZZ(factor(4))
TypeError: unable to coerce <class 'sage.structure.factorization_integer.IntegerFactorization'> to an integer
```


If we go for the coerce way, then the simplest would be to pass this
option down to `sum_of_terms`. This requires updating
`CombinatorialFreeModule.sum_of_terms` to implement that option, by
passing it down to respectively `CombinatorialFreeModule._from_dict`
where it's already implemented and `CombinatorialFreeModule.term`
where it would need to be implemented by passing the option down to
`_from_dict`.

It's getting a bit bigger of a change than what was planned for
originaly; do you feel like handling it?

Cheers,
                        Nicolas
<div id="comment:9" align="right">comment:9</div>

Replying to [@nthiery](#comment%3A7):

> If we go for the coerce way, then the simplest would be to pass this
> option down to `sum_of_terms`. This requires updating
> `CombinatorialFreeModule.sum_of_terms` to implement that option, by
> passing it down to respectively `CombinatorialFreeModule._from_dict`
> where it's already implemented and `CombinatorialFreeModule.term`
> where it would need to be implemented by passing the option down to
> `_from_dict`.

I agree and will do that as soon as my sage works again (see sage-devel thread...)
<div id="comment:10" align="right">comment:10</div>

Hi there!

I was just about to do what Nicolas suggest -- adding an optional argument `coerce`, but I encountered some things that make me hesitate:

1. to be helpful for the casual user, it's default value would have to be `True` - differently from what we have in `_from_dict`.

2. the docstring would then be something like

```
whether to try to coerce the result of applying `f` to the coefficient ring.
```

    which is a bit strange, given that we require `f` to be an endofunction.

If you reassure me, I'll do it.  I'm just not 100% sure myself.
<div id="comment:11" align="right">comment:11</div>

Replying to [@mantepse](#comment%3A10):
> 1. to be helpful for the casual user, it's default value would have to be `True`
> - differently from what we have in `_from_dict`.

Yeah, I agree; it's not supper consistent. Yet `_from_dict` is a
private method when this one is public; so it's not unreasonable
either that the typical usage situation differ, inducing different
default behavior.

> 2. the docstring would then be something like
> 
> ```
> whether to try to coerce the result of applying `f` to the coefficient ring.
> ```
> 
>     which is a bit strange, given that we require `f` to be an endofunction.

I see your point. We should be able to find some wording that makes it
sound natural. Maybe something like:

`Whether to systematically coerce to the coefficient ring the result
of applying `f`, for additional safety`.

From my perspective you can proceed! Feedback anyone else?

Cheers,
<div id="comment:12" align="right">comment:12</div>

Done (currently testing).

A question: in general, what is the requirement on the coefficients (morally, not enforced...)?

1) they should be in the base ring
2) they should be coercible to the base ring
3) you shouldn't expect anything...

1: In the internal representation, the coefficients are supposed to be
in the base ring and non zero (as tested by `bool(c)` aka
`c.__nonzero__()`).

I indeed would not be surprised if there were quite a few places in
the Sage sources where those assumptions are broken / abused. It would
be good to progressively detect and fix them. A step in this direction
is to find a better speed/safety/flexibility balance for when to run
sanity checks, or not, as you are doing here.
<div id="comment:13" align="right">comment:13</div>

As it turns out, I get several failures when I insist on trying to coerce to the base ring!  (I pushed the nonworking branch, to get feedback.) For example:

```
File "src/sage/combinat/ncsf_qsym/ncsf.py", line 3405, in sage.combinat.ncsf_qsym.ncsf.NonCommutativeSymmetricFunctions.Psi
Failed example:
    test_psi(2)
    ...
    TypeError: no conversion of this rational to integer
```

The test which fails looks as follows:

```
sage: NCSF = NonCommutativeSymmetricFunctions(ZZ)
sage: R = NCSF.R()
sage: Psi = NCSF.Psi()
sage: n = 2; a = R.sum([(-1) ** i * R[[1]*i + [n-i]] for i in range(n)])
sage: Psi(a)
```

The expected output is `Psi[2]` (literally).  The easy fix would be to add the optional argument `coerce=False` to the calls of `sum_of_terms` in `ncsf.py` everywhere, but I'm not sure whether this is what I should do - because that would mean that the answer to the question in my previous comment is "3".

Martin
<div id="comment:14" align="right">comment:14</div>

Replying to [@mantepse](#comment%3A13):
> As it turns out, I get several failures when I insist on trying to coerce to the base ring!  (I pushed the nonworking branch, to get feedback.) For example:

Ok. Can you post here the test summary with the list of failing files,
just to get an idea of how bad this is?

> ```
> File "src/sage/combinat/ncsf_qsym/ncsf.py", line 3405, in sage.combinat.ncsf_qsym.ncsf.NonCommutativeSymmetricFunctions.Psi
> Failed example:
>     test_psi(2)
>     ...
>     TypeError: no conversion of this rational to integer
> ```
> 
> The test which fails looks as follows:
> 
> ```
> sage: NCSF = NonCommutativeSymmetricFunctions(ZZ)
> sage: R = NCSF.R()
> sage: Psi = NCSF.Psi()
> sage: n = 2; a = R.sum([(-1) ** i * R[[1]*i + [n-i]] for i in range(n)])
> sage: Psi(a)
> ```
> 
> The expected output is `Psi[2]` (literally).  The easy fix would be to add the optional argument `coerce=False` to the calls of `sum_of_terms` in `ncsf.py` everywhere, but I'm not sure whether this is what I should do - because that would mean that the answer to the question in my previous comment is "3".

If some intermediate calculations involve rational coefficients, then
we definitely want a failure here. Maybe there is a way to improve the
ncsf code to better support NCSF over integers by sticking to integral
computations.

Darij, Mike, ..., what do you think? Do you have a quick way to improve NCSF's
handling of integral coefficients? Or is it ok to switch this test to
`QQ` or mark it as "not implemented" for now?
Reviewer: **Nicolas M. Thiéry**
<div id="comment:16" align="right">comment:16</div>

I think it's only

```
2 items had failures:
   3 of  21 in sage.combinat.ncsf_qsym.ncsf.NonCommutativeSymmetricFunctions.Phi.Element.verschiebung
   3 of   8 in sage.combinat.ncsf_qsym.ncsf.NonCommutativeSymmetricFunctions.Psi
    [715 tests, 6 failures, 28.15 s]
----------------------------------------------------------------------
sage -t --warn-long 97.2 src/sage/combinat/ncsf_qsym/ncsf.py  # 6 doctests failed
----------------------------------------------------------------------
```
<div id="comment:17" align="right">comment:17</div>

Out of curiosity, I modified `_from_dict` to to always try coercion.  I then get some more failures:


```
----------------------------------------------------------------------
sage -t --warn-long 74.6 src/sage/modular/modform_hecketriangle/hecke_triangle_group_element.py  # Timed out
sage -t --warn-long 74.6 src/sage/algebras/steenrod/steenrod_algebra.py  # 1 doctest failed
sage -t --warn-long 74.6 src/sage/combinat/ncsf_qsym/ncsf.py  # 6 doctests failed
sage -t --warn-long 74.6 src/sage/combinat/sf/macdonald.py  # 4 doctests failed
sage -t --warn-long 74.6 src/sage/combinat/free_module.py  # 1 doctest failed
----------------------------------------------------------------------
```
<div id="comment:18" align="right">comment:18</div>

Darij put some effort into making sure that `ncsf` and `qsym` worked over the integers in obvious ways.  I believe that this involved a lot of workarounds.

I am concerned about the conflicts with macdonald.py.  I want to check those out carefully because we are in the process of fine tuning that code in #18194

IMO: I think that if you want to allow factor coefficients then the answer to your question is 3) you shouldn't expect anything...
<div id="comment:19" align="right">comment:19</div>

Can you tell me how to exactly reproduce the failing doctests in ncsf.py? I don't have time for the rest, but at least I should see what's going wrong in my code. The test_psi(n) tests, though, have every right to fail. They should be fixed by replacing `Psi(a) == Psi[n]` by `a == R(Psi[n])`.
Changed commit from **[`3a7761e`](https://github.com/sagemath/sagetrac-mirror/commit/3a7761e95b8ca8b83e08635828a3754d751fe918)** to **[`985e716`](https://github.com/sagemath/sagetrac-mirror/commit/985e716e917594185bf6ac44f9a602f8bd92f545)**
<div id="comment:20"></div>

Branch pushed to git repo; I updated commit sha1. New commits:
<table><tr><td><a href="https://github.com/sagemath/sagetrac-mirror/commit/985e716e917594185bf6ac44f9a602f8bd92f545"><code>985e716</code></a></td><td><code>pass coerce=True down to _from_dict</code></td></tr></table>

<div id="comment:21" align="right">comment:21</div>

OOps, I thought I pushed, but I actually only pushed just now.

Darij, if you pull the current branch, you'll see the failures.
<div id="comment:22" align="right">comment:22</div>

Replying to [@zabrocki](#comment%3A18):

> IMO: I think that if you want to allow factor coefficients then the answer to your question is 3) you shouldn't expect anything...

I don't think so.  A neat trick we should copy from FriCAS is to have a constructor `Factored` that takes a ring and returns a ring, keeping the elements in factored form.

So far, however, I do not know how to create a ring in sage...
<div id="comment:23" align="right">comment:23</div>

I investigated the failure in `sage.algebras.steenrod.steenrod_algebra.SteenrodAlgebra_generic.coproduct`, which is actually not a real failure:

```
Failed example:
    SteenrodAlgebra(p=11, profile=((), (2,1,2))).Q(0,2).coproduct()
Expected:
    1 # Q_0 Q_2 + Q_0 # Q_2 + Q_0 Q_2 # 1 - Q_2 # Q_0
Got:
    1 # Q_0 Q_2 + Q_0 # Q_2 + Q_0 Q_2 # 1 + 10*Q_2 # Q_0

```

The `p=11` in the example means that the coefficient ring is the field with 11 elements, so -1 and 10 are actually the same thing.
Changed branch from **[u/mantepse/make_map_coefficients_in_modules_with_basis_safer_or_more_general](https://github.com/sagemath/sagetrac-mirror/tree/u/mantepse/make_map_coefficients_in_modules_with_basis_safer_or_more_general)** to **[public/ticket/18264](https://github.com/sagemath/sagetrac-mirror/tree/public/ticket/18264)**
Changed commit from **[`985e716`](https://github.com/sagemath/sagetrac-mirror/commit/985e716e917594185bf6ac44f9a602f8bd92f545)** to **[`d0d80e7`](https://github.com/sagemath/sagetrac-mirror/commit/d0d80e73ccae7a5823e8692e72e0c31e59800e1b)**
<div id="comment:24"></div>

New commits:
<table><tr><td><a href="https://github.com/sagemath/sagetrac-mirror/commit/6e52f396d0ea92e98f3ea0a4426ae99bd5b187ae"><code>6e52f39</code></a></td><td><code>Merge branch 'u/mantepse/make_map_coefficients_in_modules_with_basis_safer_or_more_general' of git://trac.sagemath.org/sage into map</code></td></tr><tr><td><a href="https://github.com/sagemath/sagetrac-mirror/commit/d0d80e73ccae7a5823e8692e72e0c31e59800e1b"><code>d0d80e7</code></a></td><td><code>fix ncsf.py and steenrod_algebra.py (both times the doctests were bad)</code></td></tr></table>

<div id="comment:25" align="right">comment:25</div>

I've fixed the doctests in two files.

The doc failure in macdonald.py seems a real issue. Mike? (Anyone who works on this needs the newest develop version, 6.7.beta3, since the Macdonald functions have undergone a major change in this version.)

Also the error in free_module.py itself seems to suggest that some optional parameters are being ignored...
<div id="comment:26" align="right">comment:26</div>

Replying to [@darijgr](#comment%3A25):
> I've fixed the doctests in two files.

Thank you!

> Also the error in free_module.py itself seems to suggest that some optional parameters are being ignored...

Yes, as I mentioned above (I hope), I "manually" set `coerce = True` in `_from_dict` to catch errors.
<div id="comment:27" align="right">comment:27</div>

Oh! Does this mean I've built on top of an experimental branch? Sorry!
<div id="comment:28" align="right">comment:28</div>

I'm sorry, I did not warn explicitely enough.

To get a production branch, simply remove the line `coerce = True` in `_from_dict` and recompile.
<div id="comment:29" align="right">comment:29</div>

The doctest fixes are now in #19771.
<div id="comment:30" align="right">comment:30</div>

Another case where the implementation of `map_coefficients` in `module_with_basis` is called but doesn't work:

```
sage: vector([1,2]).map_coefficients(lambda x: x+1)
...
TypeError: 'sage.rings.integer.Integer' object is not iterable
```

Resolving patch:
diff --git a/src/sage/categories/modules_with_basis.py b/src/sage/categories/modules_with_basis.py
--- a/src/sage/categories/modules_with_basis.py
+++ b/src/sage/categories/modules_with_basis.py
@@ -29,6 +29,7 @@
 from sage.categories.fields import Fields
 from sage.categories.modules import Modules
 from sage.categories.poor_man_map import PoorManMap
+from sage.categories.map import Map
 from sage.structure.element import Element, parent
 
 
@@ -174,7 +175,7 @@ def _call_(self, x):
             if M.base_ring() != self.base_ring():
                 M = M.change_ring(self.base_ring())
         except (TypeError, AttributeError) as msg:
-            raise TypeError("%s\nunable to coerce x (=%s) into %s" % (msg,x,self))
+            raise TypeError("%s\nunable to coerce x (=%s) into %s" % (msg, x, self))
         return M
 
     def is_abelian(self):
@@ -582,7 +583,7 @@ def module_morphism(self, on_basis=None, matrix=None, function=None,
                 return ModuleMorphismByLinearity(
                     domain=self, on_basis=on_basis, **keywords)
             else:
-                return ModuleMorphismFromFunction( # Or just SetMorphism?
+                return ModuleMorphismFromFunction(  # Or just SetMorphism?
                     domain=self, function=function, **keywords)
 
         _module_morphism = module_morphism
@@ -1265,7 +1266,7 @@ def _apply_module_morphism(self, x, on_basis, codomain=False):
                 except Exception:
                     raise ValueError('codomain could not be determined')
 
-            if hasattr( codomain, 'linear_combination' ):
+            if hasattr(codomain, 'linear_combination'):
                 mc = x.monomial_coefficients(copy=False)
                 return codomain.linear_combination((on_basis(key), coeff)
                                                    for key, coeff in mc.items())
@@ -1289,8 +1290,8 @@ def _apply_module_endomorphism(self, x, on_basis):
                 2*s[2, 1] + 2*s[3]
             """
             mc = x.monomial_coefficients(copy=False)
-            return self.linear_combination( (on_basis(key), coeff)
-                                            for key, coeff in mc.items())
+            return self.linear_combination((on_basis(key), coeff)
+                                           for key, coeff in mc.items())
 
         def dimension(self):
             """
@@ -2063,17 +2064,26 @@ def trailing_term(self, *args, **kwds):
             """
             return self.parent().term(*self.trailing_item(*args, **kwds))
 
-        def map_coefficients(self, f):
+        def map_coefficients(self, f, new_base_ring=None):
             """
-            Mapping a function on coefficients.
+            Return the element obtained by applying ``f`` to the non-zero
+            coefficients of ``self``.
+
+            If ``f`` is a :class:`sage.categories.map.Map`, then the resulting
+            polynomial will be defined over the codomain of ``f``. Otherwise, the
+            resulting polynomial will be over the same ring as ``self``. Set
+            ``new_base_ring`` to override this behaviour.
+
+            An error is raised if the coefficients cannot be
+            converted to the new base ring.
 
             INPUT:
 
-            - ``f`` -- an endofunction on the coefficient ring of the
-              free module
+            - ``f`` -- a callable that will be applied to the
+              coefficients of ``self``
 
-            Return a new element of ``self.parent()`` obtained by applying the
-            function ``f`` to all of the coefficients of ``self``.
+            - ``new_base_ring`` -- (optional) if given, the resulting element
+              will be defined over this ring
 
             EXAMPLES::
 
@@ -2097,8 +2107,42 @@ def map_coefficients(self, f):
                 sage: a = s([2,1]) + 2*s([3,2])                                         # needs sage.combinat sage.modules
                 sage: a.map_coefficients(lambda x: x * 2)                               # needs sage.combinat sage.modules
                 2*s[2, 1] + 4*s[3, 2]
+
+            We can map into a different base ring::
+
+                sage: F = CombinatorialFreeModule(QQ, ['a','b','c'])
+                sage: B = F.basis()
+                sage: a = 1/2*(B['a'] + 3*B['c']); a
+                1/2*B['a'] + 3/2*B['c']
+                sage: b = a.map_coefficients(lambda c: 2*c, ZZ); b
+                B['a'] + 3*B['c']
+                sage: b.parent()
+                Free module generated by {'a', 'b', 'c'} over Integer Ring
+                sage: b.map_coefficients(lambda c: 1/2*c, ZZ)
+                Traceback (most recent call last):
+                ...
+                TypeError: no conversion of this rational to integer
+
+            Coefficients are converted to the new base ring after
+            applying the map::
+
+                sage: B['a'].map_coefficients(lambda c: 2*c, GF(2))
+                0
+                sage: B['a'].map_coefficients(lambda c: GF(2)(c), QQ)
+                B['a']
+
             """
-            return self.parent().sum_of_terms( (m, f(c)) for m,c in self )
+            R = self.parent()
+            if isinstance(f, Map):
+                B = f.codomain()
+            else:
+                B = self.base_ring()
+            if new_base_ring is not None:
+                B = new_base_ring
+            if B is not self.base_ring():
+                R = R.change_ring(B)
+            mc = self.monomial_coefficients(copy=False)
+            return R.sum_of_terms((m, B(f(c))) for m, c in mc.items())
 
         def map_support(self, f):
             """
@@ -2140,7 +2184,7 @@ def map_support(self, f):
                 sage: y.parent() is B                                                   # needs sage.modules
                 True
             """
-            return self.parent().sum_of_terms( (f(m), c) for m,c in self )
+            return self.parent().sum_of_terms((f(m), c) for m, c in self)
 
         def map_support_skip_none(self, f):
             """
@@ -2174,7 +2218,9 @@ def map_support_skip_none(self, f):
                 sage: y.parent() is B                                                   # needs sage.modules
                 True
             """
-            return self.parent().sum_of_terms( (fm,c) for (fm,c) in ((f(m), c) for m,c in self) if fm is not None)
+            return self.parent().sum_of_terms((fm, c)
+                                              for fm, c in ((f(m), c) for m, c in self)
+                                              if fm is not None)
 
         def map_item(self, f):
             """
@@ -2208,7 +2254,7 @@ def map_item(self, f):
                 sage: a.map_item(f)                                                     # needs sage.combinat sage.modules
                 2*s[2, 1] + 2*s[3]
             """
-            return self.parent().sum_of_terms( f(m,c) for m,c in self )
+            return self.parent().sum_of_terms(f(m, c) for m, c in self)
 
         def tensor(*elements):
             """
@@ -2544,11 +2590,13 @@ def apply_multilinear_morphism(self, f, codomain=None):
                     except AttributeError:
                         codomain = f(*[module.zero() for module in modules]).parent()
                 if codomain in ModulesWithBasis(K):
-                    return codomain.linear_combination((f(*[module.monomial(t) for (module,t) in zip(modules, m)]), c)
-                                                       for m,c in self)
+                    return codomain.linear_combination((f(*[module.monomial(t)
+                                                            for module, t in zip(modules, m)]), c)
+                                                       for m, c in self)
                 else:
-                    return sum((c * f(*[module.monomial(t) for (module,t) in zip(modules, m)])
-                                for m,c in self),
+                    return sum((c * f(*[module.monomial(t)
+                                        for module, t in zip(modules, m)])
+                                for m, c in self),
                                codomain.zero())
 
     class DualObjects(DualObjectsCategory):


Test patch: