Repository: quantumlib/Cirq

Issue:
Cross entropy benchmarking functionality does not match documentation
Documentation:
https://github.com/quantumlib/Cirq/blob/1d35492f85aab580807709b3b012adeeda86cba7/cirq/experiments/cross_entropy_benchmarking.py#L107
Code:
https://github.com/quantumlib/Cirq/blob/1d35492f85aab580807709b3b012adeeda86cba7/cirq/experiments/cross_entropy_benchmarking.py#L335-L337

In the code, `f_meas` seems to be a quite different thing from what's documented: it is obtained by estimating the _experimental_ bitstring probabilities `prob_meas` through a experimental histogram (note that most of these will be 0 at a modest number of qubits), and then multiplying that number by the theoretical probability.

It appears that the documentation is based on the assumption of exponentially distributed (Porter-Thomas) probabilities, while the code is doing something else. Is that right?

@XiaoMiQC 


Discussion:
@viathor also to comment
@viathor @XiaoMiQC is this an issue?
I believe the formula in the documentation is actually equivalent to what is being done in the code. This has been used on actual quantum processors and I have obtained fidelity numbers as expected. One can test the equivalence by generating bits using the Cirq simulator and compute the results using the code. If it is different than what you get out of implementing the formula in the documentation, please let me know.
Circ cync: Looks like this is correct actually, @viathor will doublecheck and update status.
@XiaoMiQC @kevinsung @dstrain115 

Kevin is right. Docstring and code disagree. The former describes the linear cross-entropy benchmarking (which was our ultimate choice in QS experiment back in 2019). The fidelity estimate in this method is produced by averaging the expression `2^n p_th - 1` over the experimentally observed bitstrings (where p_th stands for the ideal/theoretical probability of each bitstring).

OTOH, the code appears to use Vadim et al's Least Mean Square for XEB wherein the fidelity estimate is computed as the ratio `(dH_measured * dH_ideal) / dH_ideal**2`. Both are known to work, but produce slightly different estimates with different statistical properties (IIRC, the former has better variance).

I don't think it makes sense to change the code, but we should update the docstring.

---

Also, it is worth noting that a number of XEB-based estimators of fidelity are implemented in [fidelity_estimation.py](https://github.com/quantumlib/Cirq/blob/master/cirq-core/cirq/experiments/fidelity_estimation.py). In particular, see [`linear_xeb_fidelity_from_probabilities`](https://github.com/quantumlib/Cirq/blob/4f881d7172b3e17084975bdc2800fe2896441c82/cirq-core/cirq/experiments/fidelity_estimation.py#L33) and [`least_squares_xeb_fidelity_from_probabilities`](https://github.com/quantumlib/Cirq/blob/4f881d7172b3e17084975bdc2800fe2896441c82/cirq-core/cirq/experiments/fidelity_estimation.py#L317) which implement the two estimators referred to above. In addition, the module implements log XEB and HOG score.

Thus, some of the XEB functionality is unfortunately duplicated. What do y'all think about removing cross_entropy_benchmarking.py? Does anyone still use this code? Could they switch to the newer and more versatile alternative in fidelity_estimation.py and related(*) modules?

(*) Note that fidelity_estimation.py has a narrower focus than cross_entropy_benchmarking.py. For example, it does not have code to generate RQCs. That said, AFAIK code to do all other XEB-related tasks exists elsewhere in cirq.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days
Issue closed due to inactivity.
for your second comment @viathor there's https://github.com/quantumlib/Cirq/issues/3775
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days

Resolving patch:
diff --git a/cirq-core/cirq/experiments/cross_entropy_benchmarking.py b/cirq-core/cirq/experiments/cross_entropy_benchmarking.py
--- a/cirq-core/cirq/experiments/cross_entropy_benchmarking.py
+++ b/cirq-core/cirq/experiments/cross_entropy_benchmarking.py
@@ -320,7 +320,7 @@ def cross_entropy_benchmarking(
     Then, take the average of $f_{mn}^{meas}$ over all circuit_mn with fixed
     n to obtain:
 
-    $f_{n} ^ {meas} = (\sum_m f_{mn}^{meas}) / M$
+    $f_{n}^{meas} = (\sum_m f_{mn}^{meas}) / M$
 
     4) Compute a theoretical XEB function for each circuit_mn:
 
@@ -332,7 +332,7 @@ def cross_entropy_benchmarking(
     Similarly, we then average $f_m^{th}$ over all circuit_mn with fixed n to
     obtain:
 
-    $f_{n} ^ {th} = (\sum_m f_{mn}^{th}) / M$
+    $f_{n}^{th} = (\sum_m f_{mn}^{th}) / M$
 
     5) Calculate the XEB fidelity $\alpha_n$ at fixed n:
 
@@ -383,7 +383,7 @@ def cross_entropy_benchmarking(
     # numbers of cycles. The values are 2D arrays with each row being the
     # probabilities obtained from a single trial.
     probs_meas = {n: np.zeros((num_circuits, 2 ** num_qubits)) for n in cycle_range}
-    probs_exp = {n: np.zeros((num_circuits, 2 ** num_qubits)) for n in cycle_range}
+    probs_th = {n: np.zeros((num_circuits, 2 ** num_qubits)) for n in cycle_range}
 
     for k in range(num_circuits):
 
@@ -402,17 +402,17 @@ def cross_entropy_benchmarking(
         # Simulate each circuit with the Cirq simulator to obtain the
         # state vector at the end of each circuit, from which the
         # theoretically expected bit-string probabilities are obtained.
-        probs_exp_k: List[np.ndarray] = []
+        probs_th_k: List[np.ndarray] = []
         for circ_k in circuits_k:
             res = simulator.simulate(circ_k, qubit_order=qubits)
             state_probs = value.state_vector_to_probabilities(np.asarray(res.final_state_vector))
-            probs_exp_k.append(state_probs)
+            probs_th_k.append(state_probs)
 
         for i, num_cycle in enumerate(cycle_range):
-            probs_exp[num_cycle][k, :] = probs_exp_k[i]
+            probs_th[num_cycle][k, :] = probs_th_k[i]
             probs_meas[num_cycle][k, :] = probs_meas_k[i]
 
-    fidelity_vals = _xeb_fidelities(probs_exp, probs_meas)
+    fidelity_vals = _xeb_fidelities(probs_th, probs_meas)
     xeb_data = [CrossEntropyPair(c, k) for (c, k) in zip(cycle_range, fidelity_vals)]
     return CrossEntropyResult(data=xeb_data, repetitions=repetitions)  # type: ignore
 
@@ -527,19 +527,35 @@ def _measure_prob_distribution(
 
 
 def _xeb_fidelities(
-    ideal_probs: Dict[int, np.ndarray], actual_probs: Dict[int, np.ndarray]
+    probs_th: Dict[int, np.ndarray], probs_meas: Dict[int, np.ndarray]
 ) -> List[float]:
-    num_cycles = sorted(list(ideal_probs.keys()))
-    return [_compute_fidelity(ideal_probs[n], actual_probs[n]) for n in num_cycles]
+    """Compute XEB fidelity estimates for all circuit depths.
+
+    Args:
+        probs_th: Theoretical output probabilities by cycle number.
+        probs_meas: Experimental output probabilities by cycle number.
+    Returns:
+        List of fidelity estimates for each circuit depth.
+    """
+    num_cycles = sorted(list(probs_th.keys()))
+    return [_compute_fidelity(probs_th[n], probs_meas[n]) for n in num_cycles]
 
 
-def _compute_fidelity(probs_exp: np.ndarray, probs_meas: np.ndarray) -> float:
-    _, num_states = probs_exp.shape
-    pp_cross = probs_exp * probs_meas
-    pp_exp = probs_exp ** 2
+def _compute_fidelity(probs_th: np.ndarray, probs_meas: np.ndarray) -> float:
+    """Compute XEB fidelity estimate.
+
+    Args:
+        probs_th: Theoretical output probabilities.
+        probs_meas: Experimental output probabilities.
+    Returns:
+        XEB fidelity estimate.
+    """
+    _, num_states = probs_th.shape
+    pp_cross = probs_th * probs_meas
+    pp_th = probs_th ** 2
     f_meas = np.mean(num_states * np.sum(pp_cross, axis=1) - 1.0)
-    f_exp = np.mean(num_states * np.sum(pp_exp, axis=1) - 1.0)
-    return float(f_meas / f_exp)
+    f_th = np.mean(num_states * np.sum(pp_th, axis=1) - 1.0)
+    return float(f_meas / f_th)
 
 
 def _random_half_rotations(qubits: Sequence[ops.Qid], num_layers: int) -> List[List[ops.OP_TREE]]:


Test patch: