Repository: Pyomo/pyomo

Issue:
Extract linear/quadratic/nonlinear terms from nonlinear expressions
## Summary

I want to extract linear/quadratic/nonlinear terms from nonlinear expressions.

### Description

<!-- Describe the solution you'd like and the alternatives you have considered. -->

I first tried `expressions.args` and check the `polynomial_degree()` of each subexpressions. However, if the subexpression is a constant. An error will be reported since `int` and `float` have no `polynomial_degree()`.

Then I tried first transforming `expressions` into `repn` and then obtaining the `linear/quadratic/nonlinear terms`. I got stuck in the second step. The `nonlinear_expr` is stored in the `repn`, but `linear_expr` and `quadratic_expr` are not stored. 

I know that `linear_coefs`, `linear_vars`, `quadratic_coefs` and `quadratic_vars` are provided. `linear_expr` and `quadratic_expr` can be obtained by `SUMPRODUCT`. 

Is there an existing function to do this? I think [`to_expression`](https://github.com/Pyomo/pyomo/blob/5683dc3d46e6c4a3a675624f23b0c14d6578be1a/pyomo/repn/standard_repn.py#L164) might be the best candidate. Can we add some args, like `linear_only`, `quadratic_only` for this function?


<!-- If you want to ask a question (how to use Pyomo, what it can do, etc.), please refer to our [online documentation](https://github.com/Pyomo/pyomo#getting-help) on ways to get help. -->



Discussion:
Yes.  See if `generate_standard_repn` will do what you want.  You will want to do some testing, though.  It is designed to gather linear and quadratic terms for the LP writer, and will "throw up its hands" and dump nonlinear things into a separate subexpression.  That said, we don't generally use it for extracting quadratic terms from a general nonlinear expression, so you will want to check that it doesn't accidentally lump some quadratic terms into the nonlinear subexpression (It shouldn't, but that is not somethign that we test).
I can confirm that `generate_standard_repn` will not always extract linear or quadratic terms from nonlinear expressions:

```
In [1]: import pyomo.environ as pe

In [2]: m = pe.ConcreteModel()

In [3]: m.x = pe.Var()

In [4]: m.y = pe.Var()

In [5]: from pyomo.repn.standard_repn import generate_standard_repn

In [6]: e = (m.x + m.y + pe.log(m.x)) * m.x

In [8]: print(generate_standard_repn(e))

constant:       0
linear vars:    []
linear var ids: []
linear coef:    []
quadratic vars:    []
quadratic var ids: []
quadratic coef:    []
nonlinear expr:
(x + y + log(x))*x
nonlinear vars: ['x', 'y']



In [9]: e = (m.x + m.y + pe.log(m.x) + 2) * m.x

In [10]: print(generate_standard_repn(e))

constant:       0
linear vars:    []
linear var ids: []
linear coef:    []
quadratic vars:    []
quadratic var ids: []
quadratic coef:    []
nonlinear expr:
(x + y + log(x) + 2)*x
nonlinear vars: ['x', 'y']
```
OK.  I am not too surprised.  I have been working on new versions of the expression compilers (replacements for `generate_standard_repn`) that walk the expressions from the "bottom up" instead of "top down".  They should help make the routines more robust.  Unfortunately, at the moment both those walkers (sitting on the `nl-writer-cache-expressions` and `templated-lp-models` branches of my fork) only extract the linear portion of the models.  I have a reasonable idea how to extend them to gathering quadratic terms (for the LP writers), but that is something I probably won't get to for several months.
Thanks. @jsiirola @michaelbynum 
I have one question here. 
If the coefficient of linear terms is `native_numeric_types`, why would the accumulation become so complicated? like `expr -= - c*v`.

https://github.com/Pyomo/pyomo/blob/5683dc3d46e6c4a3a675624f23b0c14d6578be1a/pyomo/repn/standard_repn.py#L180-L188


I would argue that this issue could be kept open as a placeholder for the walker that Jhon is putting together. It would certainly make Pyomo's life easier in the upcoming world where more and more (MI)LP solvers are supporting quadratic constraints.
@bernalde, agreed.  Note that the LP writer and `generate_standard_repn` work fine for quadratic expressions.  The limitation @michaelbynum pointed out (which I am not surprised at) is that `generate_standard_repn` will not reliability identify / extract quadratic terms from general nonlinear expressions.
Well said, @jsiirola. Thank you for clarifying that.

Resolving patch:
diff --git a/pyomo/repn/quadratic.py b/pyomo/repn/quadratic.py
--- a/pyomo/repn/quadratic.py
+++ b/pyomo/repn/quadratic.py
@@ -222,7 +222,6 @@ def _handle_product_nonlinear(visitor, node, arg1, arg2):
     # We are multiplying (A + Bx + Cx^2 + D(x)) * (A + Bx + Cx^2 + Dx))
     _, x1 = arg1
     _, x2 = arg2
-    ans = visitor.Result()
     ans.multiplier = x1.multiplier * x2.multiplier
     x1.multiplier = x2.multiplier = 1
     # x1.const * x2.const [AA]
@@ -274,7 +273,7 @@ def _handle_product_nonlinear(visitor, node, arg1, arg2):
         x1.quadratic = None
     x2.linear = {}
     # [BC] + [BD]
-    if x1_lin:
+    if x1_lin and (x2.nonlinear is not None or x2.quadratic):
         x1.linear = x1_lin
         ans.nonlinear += x1.to_expression(visitor) * x2.to_expression(visitor)
     # [AD]
@@ -294,6 +293,9 @@ def _handle_product_nonlinear(visitor, node, arg1, arg2):
         (_QUADRATIC, _GENERAL): _handle_product_nonlinear,
         # Replace handler from the linear walker
         (_LINEAR, _LINEAR): _handle_product_linear_linear,
+        (_GENERAL, _GENERAL): _handle_product_nonlinear,
+        (_GENERAL, _LINEAR): _handle_product_nonlinear,
+        (_LINEAR, _GENERAL): _handle_product_nonlinear,
     }
 )
 


Test patch:
diff --git a/pyomo/repn/tests/test_quadratic.py b/pyomo/repn/tests/test_quadratic.py
--- a/pyomo/repn/tests/test_quadratic.py
+++ b/pyomo/repn/tests/test_quadratic.py
@@ -126,8 +126,6 @@ def test_product(self):
         self.assertEqual(repn.constant, 0)
         self.assertEqual(repn.linear, {})
         self.assertEqual(repn.quadratic, None)
-        print(repn.nonlinear)
-        print(NL)
         assertExpressionsEqual(self, repn.nonlinear, NL)
 
         e = (1 + 2 * m.x + 3 * m.y) * (4 + 5 * m.x + 6 * m.y)
@@ -147,6 +145,70 @@ def test_product(self):
         )
         assertExpressionsEqual(self, repn.nonlinear, None)
 
+        e = (m.x + m.y + log(m.x)) * m.x
+
+        cfg = VisitorConfig()
+        visitor = QuadraticRepnVisitor(*cfg)
+        visitor.expand_nonlinear_products = False
+        repn = visitor.walk_expression(e)
+
+        NL = (log(m.x) + (m.x + m.y)) * m.x
+
+        self.assertEqual(cfg.subexpr, {})
+        self.assertEqual(cfg.var_map, {id(m.x): m.x, id(m.y): m.y})
+        self.assertEqual(cfg.var_order, {id(m.x): 0, id(m.y): 1})
+        self.assertEqual(repn.multiplier, 1)
+        self.assertEqual(repn.constant, 0)
+        self.assertEqual(repn.linear, {})
+        self.assertEqual(repn.quadratic, None)
+        assertExpressionsEqual(self, repn.nonlinear, NL)
+
+        visitor.expand_nonlinear_products = True
+        repn = visitor.walk_expression(e)
+
+        NL = log(m.x) * m.x
+
+        self.assertEqual(cfg.subexpr, {})
+        self.assertEqual(cfg.var_map, {id(m.x): m.x, id(m.y): m.y})
+        self.assertEqual(cfg.var_order, {id(m.x): 0, id(m.y): 1})
+        self.assertEqual(repn.multiplier, 1)
+        self.assertEqual(repn.constant, 0)
+        self.assertEqual(repn.linear, {})
+        self.assertEqual(repn.quadratic, {(id(m.x), id(m.x)): 1, (id(m.x), id(m.y)): 1})
+        assertExpressionsEqual(self, repn.nonlinear, NL)
+
+        e = m.x * (m.x + m.y + log(m.x) + 2)
+
+        cfg = VisitorConfig()
+        visitor = QuadraticRepnVisitor(*cfg)
+        visitor.expand_nonlinear_products = False
+        repn = visitor.walk_expression(e)
+
+        NL = m.x * (log(m.x) + (m.x + m.y) + 2)
+
+        self.assertEqual(cfg.subexpr, {})
+        self.assertEqual(cfg.var_map, {id(m.x): m.x, id(m.y): m.y})
+        self.assertEqual(cfg.var_order, {id(m.x): 0, id(m.y): 1})
+        self.assertEqual(repn.multiplier, 1)
+        self.assertEqual(repn.constant, 0)
+        self.assertEqual(repn.linear, {})
+        self.assertEqual(repn.quadratic, None)
+        assertExpressionsEqual(self, repn.nonlinear, NL)
+
+        visitor.expand_nonlinear_products = True
+        repn = visitor.walk_expression(e)
+
+        NL = m.x * log(m.x)
+
+        self.assertEqual(cfg.subexpr, {})
+        self.assertEqual(cfg.var_map, {id(m.x): m.x, id(m.y): m.y})
+        self.assertEqual(cfg.var_order, {id(m.x): 0, id(m.y): 1})
+        self.assertEqual(repn.multiplier, 1)
+        self.assertEqual(repn.constant, 0)
+        self.assertEqual(repn.linear, {id(m.x): 2})
+        self.assertEqual(repn.quadratic, {(id(m.x), id(m.x)): 1, (id(m.x), id(m.y)): 1})
+        assertExpressionsEqual(self, repn.nonlinear, NL)
+
     def test_sum(self):
         m = ConcreteModel()
         m.x = Var()