Repository: Qiskit/qiskit

Issue:
UMDA optimizer incorrectly logs objective function values
### Environment

- **Qiskit Terra version**: 0.21.0
- **Python version**: 3.8.13
- **Operating system**: Ubuntu 20.04.4 LTS


### What is happening?

I have noticed that results returned by UMDA optimizer via a callback wildly oscillate and yet the optimizer consistently and progressively finds the optimum value at the end.

Working with the source of UMDA class, I have noticed that the class uses the concept of "generation" in which it performs numerous stochastic objective function evaluations and at the end identifies the best which should be reported via a callback as the next value of the optimizer iteration. However, UMDA uses the minimize() function for all within-generation evaluations, and this in turn triggers callbacks, which report both the generation-optimum value and all other intermediate evaluations, which cause the wild "plot" oscillations - this will be a problem with understanding the optimization or later publishing such experiments. This may be difficult to correct as _check_generation() relies on the externally provided objective function, which for all TrainableModels (such as NeuralNetworkRegressor) already hard-codes the callback within (when specified). 

In the interim, it is possible to make a small change to UMDA to allow the user a non-standard access to the "history" variable, which collects all generational minima, the values that should have been reported via callbacks. At the moment the "history" variable is local and cannot be accessed by the user. All that is required is to make "history" an object variable (turn it into "self.history" in three places). When suitably documented, the user callback function could then watch the changes to the length of the history list and collect or use the callback argument values as needed.

I attach two charts, one INCORRECT produced from the standard callbacks plotting the objective function values after each iteration of UMDA (including the within-generation evaluations) and the second which is CORRECT plotting only the values logged in the UMDA history list.

![UMDA_error_objective_fun](https://user-images.githubusercontent.com/6309054/188521341-7d9f573a-d2ed-41ff-bbce-4854b75bfe34.png)

![UMDA_history_objective_fun png](https://user-images.githubusercontent.com/6309054/188521360-efe3cac1-1a20-467d-abe8-3972b8b8dbb1.png)



### How can we reproduce the issue?

Here is the regressor "standard" callback, which produces incorrect values of the objective function.
```
# Create a callback class
class Regr_callback:
    name = "Regr_callback"
    
    def __init__(self):
        self.objfun_vals = []

    def plot(self):
        clear_output(wait=True)
        plt.rcParams["figure.figsize"] = (12, 6)
        plt.title("Objective function")
        plt.xlabel("Iteration")
        plt.ylabel("Objective function value")
        plt.plot(range(len(self.objfun_vals)), self.objfun_vals)
        plt.show()

    # Store objective function values and weights and plot every iteration
    def graph(self, weights, obj_func_eval):
        self.objfun_vals.append(obj_func_eval)
        self.plot()

# Select UMDA optimizer and a callback function
optimizer = UMDA(maxiter=50, size_gen=30, alpha = 0.6)
regr_callback = Regr_callback()

# Define a regressor
regressor = NeuralNetworkRegressor(
    neural_network=regr_qnn,
    loss="squared_error",
    optimizer=optimizer, 
    callback=regr_callback.graph
)

# Fit model to data and plot progress, select some data for X and y
regressor.fit(X, y) 
```

### What should happen?

Should the issue be properly fixed, the above code will produce the correct plot. If not then make the "history" an object variable, in which case the callback function can be rewritten as follows, so that it works properly for all kinds of optimizers. 
```
class Regr_callback:
    name = "Regr_callback"
    
    # For use with UMDA optimizer, the optimizer needs to be passed in
    def __init__(self, umda_optimizer=None):
        self.objfun_vals = []

    def plot(self):
        clear_output(wait=True)
        plt.rcParams["figure.figsize"] = (12, 6)
        plt.title("Objective function")
        plt.xlabel("Iteration")
        plt.ylabel("Objective function value")
        plt.plot(range(len(self.objfun_vals)), self.objfun_vals)
        plt.show()

    # Store objective function values and weights and plot every evaluation, incl. within-generation
    def graph_all_evals(self, weights, obj_func_eval):
        self.objfun_vals.append(obj_func_eval)
        self.plot()

    # Store objective function values and weights and plot
    # As UMDA calls every iteration, including the internal evaluations, 
    # the obj function values must be taken from the optimizer _history
    def graph(self, weights, obj_func_eval):
        if self.umda_optimizer == None:
            self.objfun_vals.append(obj_func_eval)
            self.plot()
        elif len(self.umda_optimizer._history) > len(self.objfun_vals):
            # We do not know in which order events are handled so take it from history
            self.objfun_vals.append(self.umda_optimizer.history[-1])
            self.plot()
```
When using the callback with UMDA an optimizer will have to be passed in on the creation of the callback, e.g.
```
# Select UMDA optimizer and a callback function
optimizer = UMDA(maxiter=50, size_gen=30, alpha = 0.6)
regr_callback = Regr_callback(umda_optimizer=optimizer)
```

### Any suggestions?

All explained above.


Discussion:
I will handle this issue

Resolving patch:
diff --git a/qiskit/algorithms/optimizers/umda.py b/qiskit/algorithms/optimizers/umda.py
--- a/qiskit/algorithms/optimizers/umda.py
+++ b/qiskit/algorithms/optimizers/umda.py
@@ -11,6 +11,7 @@
 # that they have been altered from the originals.
 
 """Univariate Marginal Distribution Algorithm (Estimation-of-Distribution-Algorithm)."""
+
 from __future__ import annotations
 
 from collections.abc import Callable
@@ -122,12 +123,21 @@ class UMDA(Optimizer):
     ELITE_FACTOR = 0.4
     STD_BOUND = 0.3
 
-    def __init__(self, maxiter: int = 100, size_gen: int = 20, alpha: float = 0.5) -> None:
+    def __init__(
+        self,
+        maxiter: int = 100,
+        size_gen: int = 20,
+        alpha: float = 0.5,
+        callback: Callable[[int, np.array, float], None] | None = None,
+    ) -> None:
         r"""
         Args:
             maxiter: Maximum number of iterations.
             size_gen: Population size of each generation.
             alpha: Percentage (0, 1] of the population to be selected as elite selection.
+            callback: A callback function passed information in each iteration step. The
+                information is, in this order: the number of function evaluations, the parameters,
+                the best function value in this iteration.
         """
 
         self.size_gen = size_gen
@@ -148,6 +158,8 @@ def __init__(self, maxiter: int = 100, size_gen: int = 20, alpha: float = 0.5) -
 
         self._n_variables: int | None = None
 
+        self.callback = callback
+
     def _initialization(self) -> np.ndarray:
         vector = np.zeros((4, self._n_variables))
 
@@ -243,6 +255,11 @@ def minimize(
                 if not_better_count >= self._dead_iter:
                     break
 
+            if self.callback is not None:
+                self.callback(
+                    len(history) * self._size_gen, self._best_ind_global, self._best_cost_global
+                )
+
             self._new_generation()
 
         result.x = self._best_ind_global
@@ -321,6 +338,7 @@ def settings(self) -> dict[str, Any]:
             "maxiter": self.maxiter,
             "alpha": self.alpha,
             "size_gen": self.size_gen,
+            "callback": self.callback,
         }
 
     def get_support_level(self):


Test patch:
diff --git a/test/python/algorithms/optimizers/test_umda.py b/test/python/algorithms/optimizers/test_umda.py
--- a/test/python/algorithms/optimizers/test_umda.py
+++ b/test/python/algorithms/optimizers/test_umda.py
@@ -14,6 +14,8 @@
 
 from test.python.algorithms import QiskitAlgorithmsTestCase
 
+import numpy as np
+
 from qiskit.algorithms.optimizers.umda import UMDA
 
 
@@ -45,6 +47,7 @@ def test_settings(self):
             "maxiter": 100,
             "alpha": 0.6,
             "size_gen": 30,
+            "callback": None,
         }
 
         assert umda.settings == set_
@@ -52,7 +55,6 @@ def test_settings(self):
     def test_minimize(self):
         """optimize function test"""
         from scipy.optimize import rosen
-        import numpy as np
         from qiskit.utils import algorithm_globals
 
         # UMDA is volatile so we need to set the seeds for the execution
@@ -66,3 +68,29 @@ def test_minimize(self):
         assert len(res.x) == len(x_0)
 
         np.testing.assert_array_almost_equal(res.x, [1.0] * len(x_0), decimal=2)
+
+    def test_callback(self):
+        """Test the callback."""
+
+        def objective(x):
+            return np.linalg.norm(x) - 1
+
+        nfevs, parameters, fvals = [], [], []
+
+        def store_history(*args):
+            nfevs.append(args[0])
+            parameters.append(args[1])
+            fvals.append(args[2])
+
+        optimizer = UMDA(maxiter=1, callback=store_history)
+        _ = optimizer.minimize(objective, x0=np.arange(5))
+
+        self.assertEqual(len(nfevs), 1)
+        self.assertIsInstance(nfevs[0], int)
+
+        self.assertEqual(len(parameters), 1)
+        self.assertIsInstance(parameters[0], np.ndarray)
+        self.assertEqual(parameters[0].size, 5)
+
+        self.assertEqual(len(fvals), 1)
+        self.assertIsInstance(fvals[0], float)