Repository: Qiskit/qiskit

Issue:
Callback function in VQD is defined but never used
### What should we add?

The Variational Quantum Deflation algorithm has an attribute `self._callback` which is never used in `vqd.solve()`.
Inspired by the code for the VQE algorithm, the callback should be called in the `energy_evaluation()` call.
It also seems interesting that the callback also informs the user of the current step (VQD runs k steps of VQE).

Currently, the test `test_callback` checks the equality of empty lists
```
history = {'eval_count': [], 'parameters': [], 'mean': [], 'std': []}
```

The change would consist in adding the following lines during the call of the `energy_evaluation()`. These lines were taken from the VQE algorithm.
```
if self._callback is not None:
    variance = np.real(expectation.compute_variance(sampled_expect_op))
    estimator_error = np.sqrt(variance / self.quantum_instance.run_config.shots)
    for i, param_set in enumerate(parameter_sets):
        self._eval_count += 1
        self._callback(self._eval_count, param_set, means[i], estimator_error[i], step)
else:
    self._eval_count += len(means)
```
Giving
```
history {
'eval_count': [1, 2, 3, 1, 2, 3],
'parameters': [array([ 3.61186007,  4.19301252,  0.6019852 ,  5.94953681, -3.30704704,
        1.84629318, -5.4660436 ,  0.6984088 ]), array([ 4.61186007,  4.19301252,  0.6019852 ,  5.94953681, -3.30704704,
        1.84629318, -5.4660436 ,  0.6984088 ]), array([ 4.61186007,  5.19301252,  0.6019852 ,  5.94953681, -3.30704704,
        1.84629318, -5.4660436 ,  0.6984088 ]), array([ 3.61186007,  4.19301252,  0.6019852 ,  5.94953681, -3.30704704,
        1.84629318, -5.4660436 ,  0.6984088 ]), array([ 4.61186007,  4.19301252,  0.6019852 ,  5.94953681, -3.30704704,
        1.84629318, -5.4660436 ,  0.6984088 ]), array([ 3.61186007,  5.19301252,  0.6019852 ,  5.94953681, -3.30704704,
        1.84629318, -5.4660436 ,  0.6984088 ])], 
'mean': [-1.0632369218371718, -1.4565924844031897, -1.360034418572368, 37.340083390662826,        48.54340751559681, 28.58589345125128], 
'std': [0.011432088966174363, 0.009556915198516208, 0.013956731222284432, 0.011432088966174363, 0.009556915198516208, 0.015443719747907501], 
'step': [1, 1, 1, 2, 2, 2]
}

```
[WIP] Qobj class, schema validation, and update backends to accept qobj
<!--- Provide a general summary of your changes in the Title above -->
This is a replacement for two previous PRs (#144 and #326) that attempted to implement features related to qobj. In light of recent changes to the qiskit core, and recent changes to the qobj spec itslef, we are going to attempt to integrate qobj as an object in the qiskit flow, pass a serialized form of it to backends, and validate them against updated schemas.

@diego-plan9 and @ewinston can you please help port over the files from your previous PRs that we should have here? @atilag can you please ensure the jobprocessor and tests are up to date? I will work on the schemas and parsing of the results received from backends.

## Description
<!--- Describe your changes in detail -->

## Motivation and Context
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->

## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->

## Screenshots (if appropriate):

## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.


Discussion:

Closing this WIP in order to issue a new one without conflicts and with only the `QObj` implementation.

Resolving patch:
diff --git a/qiskit/algorithms/eigen_solvers/vqd.py b/qiskit/algorithms/eigen_solvers/vqd.py
--- a/qiskit/algorithms/eigen_solvers/vqd.py
+++ b/qiskit/algorithms/eigen_solvers/vqd.py
@@ -108,10 +108,11 @@ def __init__(
         Args:
             ansatz: A parameterized circuit used as ansatz for the wave function.
             k: the number of eigenvalues to return. Returns the lowest k eigenvalues.
-            betas: beta parameter in the VQD paper. Should have size k -1, the number of excited states.
-                It is a hyperparameter that balances the contribution of the overlap
-                term to the cost function and has a default value computed as
-                mean square sum of coefficients of observable.
+            betas: beta parameters in the VQD paper.
+                Should have length k - 1, with k the number of excited states.
+                These hyperparameters balance the contribution of each overlap term to the cost
+                function and have a default value computed as the mean square sum of the
+                coefficients of the observable.
             optimizer: A classical optimizer. Can either be a Qiskit optimizer or a callable
                 that takes an array as input and returns a Qiskit or SciPy optimization result.
             initial_point: An optional initial point (i.e. initial parameter values)
@@ -141,8 +142,8 @@ def __init__(
             callback: a callback that can access the intermediate data during the optimization.
                 Four parameter values are passed to the callback as follows during each evaluation
                 by the optimizer for its current set of parameters as it works towards the minimum.
-                These are: the evaluation count, the optimizer parameters for the
-                ansatz, the evaluated mean and the evaluated standard deviation.`
+                These are: the evaluation count, the optimizer parameters for the ansatz, the
+                evaluated mean, the evaluated standard deviation, and the current step.
             quantum_instance: Quantum Instance or Backend
 
         """
@@ -264,12 +265,12 @@ def include_custom(self, include_custom: bool):
             self.expectation = None
 
     @property
-    def callback(self) -> Optional[Callable[[int, np.ndarray, float, float], None]]:
+    def callback(self) -> Optional[Callable[[int, np.ndarray, float, float, int], None]]:
         """Returns callback"""
         return self._callback
 
     @callback.setter
-    def callback(self, callback: Optional[Callable[[int, np.ndarray, float, float], None]]):
+    def callback(self, callback: Optional[Callable[[int, np.ndarray, float, float, int], None]]):
         """Sets callback"""
         self._callback = callback
 
@@ -475,7 +476,8 @@ def _eval_aux_ops(
         aux_op_results = zip(aux_op_means, std_devs)
 
         # Return None eigenvalues for None operators if aux_operators is a list.
-        # None operators are already dropped in compute_minimum_eigenvalue if aux_operators is a dict.
+        # None operators are already dropped in compute_minimum_eigenvalue if aux_operators is a
+        # dict.
         if isinstance(aux_operators, list):
             aux_operator_eigenvalues = [None] * len(aux_operators)
             key_value_iterator = enumerate(aux_op_results)
@@ -608,7 +610,8 @@ def compute_eigenvalues(
             if step == 1:
 
                 logger.info(
-                    "Ground state optimization complete in %s seconds.\nFound opt_params %s in %s evals",
+                    "Ground state optimization complete in %s seconds.\n"
+                    "Found opt_params %s in %s evals",
                     eval_time,
                     result.optimal_point,
                     self._eval_count,
@@ -616,7 +619,8 @@ def compute_eigenvalues(
             else:
                 logger.info(
                     (
-                        "%s excited state optimization complete in %s s.\nFound opt_parms %s in %s evals"
+                        "%s excited state optimization complete in %s s.\n"
+                        "Found opt_params %s in %s evals"
                     ),
                     str(step - 1),
                     eval_time,
@@ -624,7 +628,7 @@ def compute_eigenvalues(
                     self._eval_count,
                 )
 
-        # To match the siignature of NumpyEigenSolver Result
+        # To match the signature of NumpyEigenSolver Result
         result.eigenstates = ListOp([StateFn(vec) for vec in result.eigenstates])
         result.eigenvalues = np.array(result.eigenvalues)
         result.optimal_point = np.array(result.optimal_point)
@@ -649,7 +653,7 @@ def get_energy_evaluation(
         This return value is the objective function to be passed to the optimizer for evaluation.
 
         Args:
-            step: level of enegy being calculated. 0 for ground, 1 for first excited state and so on.
+            step: level of energy being calculated. 0 for ground, 1 for first excited state...
             operator: The operator whose energy to evaluate.
             return_expectation: If True, return the ``ExpectationBase`` expectation converter used
                 in the construction of the expectation value. Useful e.g. to evaluate other
@@ -695,22 +699,29 @@ def get_energy_evaluation(
 
         def energy_evaluation(parameters):
             parameter_sets = np.reshape(parameters, (-1, num_parameters))
-            # Create dict associating each parameter with the lists of parameterization values for it
+            # Dict associating each parameter with the lists of parameterization values for it
             param_bindings = dict(zip(ansatz_params, parameter_sets.transpose().tolist()))
 
             sampled_expect_op = self._circuit_sampler.convert(expect_op, params=param_bindings)
-            mean = np.real(sampled_expect_op.eval())
+            means = np.real(sampled_expect_op.eval())
 
             for state in range(step - 1):
                 sampled_final_op = self._circuit_sampler.convert(
                     overlap_op[state], params=param_bindings
                 )
                 cost = sampled_final_op.eval()
-                mean += np.real(self.betas[state] * np.conj(cost) * cost)
-
-            self._eval_count += len(mean)
+                means += np.real(self.betas[state] * np.conj(cost) * cost)
+
+            if self._callback is not None:
+                variance = np.real(expectation.compute_variance(sampled_expect_op))
+                estimator_error = np.sqrt(variance / self.quantum_instance.run_config.shots)
+                for i, param_set in enumerate(parameter_sets):
+                    self._eval_count += 1
+                    self._callback(self._eval_count, param_set, means[i], estimator_error[i], step)
+            else:
+                self._eval_count += len(means)
 
-            return mean if len(mean) > 1 else mean[0]
+            return means if len(means) > 1 else means[0]
 
         if return_expectation:
             return energy_evaluation, expectation


Test patch:
diff --git a/test/python/algorithms/test_vqd.py b/test/python/algorithms/test_vqd.py
--- a/test/python/algorithms/test_vqd.py
+++ b/test/python/algorithms/test_vqd.py
@@ -257,13 +257,14 @@ def test_with_aer_qasm_snapshot_mode(self):
 
     def test_callback(self):
         """Test the callback on VQD."""
-        history = {"eval_count": [], "parameters": [], "mean": [], "std": []}
+        history = {"eval_count": [], "parameters": [], "mean": [], "std": [], "step": []}
 
-        def store_intermediate_result(eval_count, parameters, mean, std):
+        def store_intermediate_result(eval_count, parameters, mean, std, step):
             history["eval_count"].append(eval_count)
             history["parameters"].append(parameters)
             history["mean"].append(mean)
             history["std"].append(std)
+            history["step"].append(step)
 
         optimizer = COBYLA(maxiter=3)
         wavefunction = self.ry_wavefunction
@@ -279,9 +280,20 @@ def store_intermediate_result(eval_count, parameters, mean, std):
         self.assertTrue(all(isinstance(count, int) for count in history["eval_count"]))
         self.assertTrue(all(isinstance(mean, float) for mean in history["mean"]))
         self.assertTrue(all(isinstance(std, float) for std in history["std"]))
+        self.assertTrue(all(isinstance(count, int) for count in history["step"]))
         for params in history["parameters"]:
             self.assertTrue(all(isinstance(param, float) for param in params))
 
+        ref_eval_count = [1, 2, 3, 1, 2, 3]
+        ref_mean = [-1.063, -1.457, -1.360, 37.340, 48.543, 28.586]
+        ref_std = [0.011, 0.010, 0.014, 0.011, 0.010, 0.015]
+        ref_step = [1, 1, 1, 2, 2, 2]
+
+        np.testing.assert_array_almost_equal(history["eval_count"], ref_eval_count, decimal=0)
+        np.testing.assert_array_almost_equal(history["mean"], ref_mean, decimal=2)
+        np.testing.assert_array_almost_equal(history["std"], ref_std, decimal=2)
+        np.testing.assert_array_almost_equal(history["step"], ref_step, decimal=0)
+
     def test_reuse(self):
         """Test re-using a VQD algorithm instance."""
         vqd = VQD(k=1)