Repository: PennyLaneAI/pennylane-lightning
Pull request #1104: Use cuquantum API for computing expval(pauli_string) for LGPU
Author: login: josephleekl
Labels: name: ci:use-multi-gpu-runner
description: Enable usage of Multi-GPU runner for this Pull Request
name: ci:use-gpu-runner
description: Enable usage of GPU runner for this Pull Request
Created: 2025-03-19T21:15:56Z | Merged: 2025-03-24T16:38:38Z

### Before submitting

Please complete the following checklist when submitting a PR:

- [ ] All new features must include a unit test.
      If you've fixed a bug or added code that should be tested, add a test to the
      [`tests`](../tests) directory!

- [ ] All new functions and code must be clearly commented and documented.
      If you do make documentation changes, make sure that the docs build and
      render correctly by running `make docs`.

- [ ] Ensure that the test suite passes, by running `make test`.

- [ ] Add a new entry to the `.github/CHANGELOG.md` file, summarizing the
      change, and including a link back to the PR.

- [ ] Ensure that code is properly formatted by running `make format`. 

When all the above are checked, delete everything above the dashed
line and fill in the pull request template.

------------------------------------------------------------------------------------------------------------

**Context:**
Currently when calling `expval(pauli_string)` in LGPU, we use the general matrix observable computation. However, the native C++ function to call `expval(pauli_string)` is already available (which uses `custatevecComputeExpectationsOnPauliBasis`), but not called from the Python layer.

**Description of the Change:**
For LGPU, if the observable in an expval is a pauli string, we call the native C++ function.

**Benefits:**
Significant speedup for expectation value calculation of large hamiltonians. For example

| Molecule | Qubits | Num terms in hamiltonian | timing of master branch (s) | timing of updated branch (s) |
|----------|--------|--------------------------|-----------------------------|------------------------------|
| H10      |     20 |                     7151 |                        2.62 | 1.34                        |
| CO2      | 30     | 16170                    | 5599.27                     | 60.4                       |

**Possible Drawbacks:**

**Related GitHub Issues:**

[sc-85843]


Diff:
diff --git a/.github/CHANGELOG.md b/.github/CHANGELOG.md
index 7c7060c311..3d5bda5f36 100644
--- a/.github/CHANGELOG.md
+++ b/.github/CHANGELOG.md
@@ -8,6 +8,9 @@
 
 ### Improvements
 
+* Use cuquantum API for calculating expectation value of Pauli sentences in Lightning-GPU.
+  [(#1104)](https://github.com/PennyLaneAI/pennylane-lightning/pull/1104)
+
 * Hide internal C++ APIs in Lightning docs.
   [(#1096)](https://github.com/PennyLaneAI/pennylane-lightning/pull/1096)
 
diff --git a/pennylane_lightning/core/_version.py b/pennylane_lightning/core/_version.py
index 29486fbc19..60a4873807 100644
--- a/pennylane_lightning/core/_version.py
+++ b/pennylane_lightning/core/_version.py
@@ -16,4 +16,4 @@
 Version number (major.minor.patch[-label])
 """
 
-__version__ = "0.41.0-dev44"
+__version__ = "0.41.0-dev45"
diff --git a/pennylane_lightning/core/src/observables/Observables.hpp b/pennylane_lightning/core/src/observables/Observables.hpp
index 2b80770ce1..072b502e51 100644
--- a/pennylane_lightning/core/src/observables/Observables.hpp
+++ b/pennylane_lightning/core/src/observables/Observables.hpp
@@ -698,4 +698,4 @@ class SparseHamiltonianBase : public Observable<StateVectorT> {
     };
 };
 
-} // namespace Pennylane::Observables
\ No newline at end of file
+} // namespace Pennylane::Observables
diff --git a/pennylane_lightning/lightning_gpu/_measurements.py b/pennylane_lightning/lightning_gpu/_measurements.py
index ef328abbe1..aa9c6d2b63 100644
--- a/pennylane_lightning/lightning_gpu/_measurements.py
+++ b/pennylane_lightning/lightning_gpu/_measurements.py
@@ -173,6 +173,13 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            pwords, coeffs = zip(*measurementprocess.obs.pauli_rep.items())
+            pauli_words = [qml.pauli.pauli_word_to_string(p) for p in pwords]
+            wires = [p.wires.tolist() for p in pwords]
+            return self._measurement_lightning.expval(pauli_words, wires, coeffs)
+
         # use specialized functors to compute expval(Hermitian)
         if isinstance(measurementprocess.obs, qml.Hermitian):
             observable_wires = measurementprocess.obs.wires
diff --git a/tests/lightning_base/test_expval.py b/tests/lightning_base/test_expval.py
index ad30922be8..a042c4217d 100644
--- a/tests/lightning_base/test_expval.py
+++ b/tests/lightning_base/test_expval.py
@@ -384,3 +384,57 @@ def test_PauliZ_hadamard_PauliY(self, theta, phi, varphi, dev, tol):
         tol = 1e-5 if dev.c_dtype == np.complex64 else 1e-7
 
         assert np.allclose(calculated_val, reference_val, tol)
+
+    def test_permute_order(self, theta, phi, varphi, dev, tol):
+        """Tests a tensor product with permuted term order."""
+
+        subroutine = [
+            qml.RX(theta, wires=[0]),
+            qml.RX(phi, wires=[1]),
+            qml.RX(varphi, wires=[2]),
+            qml.CNOT(wires=[0, 1]),
+            qml.CNOT(wires=[1, 2]),
+        ]
+
+        ob1 = qml.PauliZ(0) @ qml.X(1) @ qml.PauliY(2)
+        ob2 = qml.PauliY(2) @ qml.X(1) @ qml.PauliZ(0)
+
+        tape1 = qml.tape.QuantumScript(subroutine, [qml.expval(ob1)])
+        tape2 = qml.tape.QuantumScript(subroutine, [qml.expval(ob2)])
+
+        calculated_val_1 = process_and_execute(dev, tape1)
+        calculated_val_2 = process_and_execute(dev, tape2)
+        reference_val = calculate_reference(tape1)
+
+        tol = 1e-5 if dev.c_dtype == np.complex64 else 1e-7
+
+        assert np.allclose(calculated_val_1, calculated_val_2, tol)
+        assert np.allclose(calculated_val_1, reference_val, tol)
+
+    def test_hamiltonian(self, theta, phi, varphi, dev, tol):
+        """Tests the expectation value of a Hamiltonian with multiple pauli words and coefficients."""
+
+        subroutine = [
+            qml.RX(theta, wires=[0]),
+            qml.RX(phi, wires=[1]),
+            qml.RX(varphi, wires=[2]),
+            qml.CNOT(wires=[0, 1]),
+            qml.CNOT(wires=[1, 2]),
+        ]
+
+        ob1 = qml.PauliZ(0) @ qml.X(1) @ qml.PauliY(2)
+        ob2 = qml.PauliY(0) @ qml.Z(1) @ qml.PauliX(2)
+        ob3 = qml.Identity(0) @ qml.X(1)
+        ob4 = qml.PauliY(2) @ qml.Z(1)
+        ob5 = qml.PauliZ(2)
+
+        tape = qml.tape.QuantumScript(
+            subroutine, [qml.expval(0.3 * ob1 + 0.4 * ob2 + 0.5 * ob3 + 0.6 * ob4 + 0.7 * ob5)]
+        )
+
+        calculated_val = process_and_execute(dev, tape)
+        reference_val = calculate_reference(tape)
+
+        tol = 1e-5 if dev.c_dtype == np.complex64 else 1e-7
+
+        assert np.allclose(calculated_val, reference_val, tol)


Reviews:
user: login: josephleekl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T20:11:22Z
user: login: LuisAlfredoNu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T21:03:53Z
user: login: josephleekl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T21:21:22Z
user: login: multiphaseCFD
body: Nice work @josephleekl ! Just a couple of Qs.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T21:21:32Z
user: login: josephleekl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T22:23:03Z
user: login: josephleekl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T22:23:29Z
user: login: josephleekl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T22:23:41Z
user: login: multiphaseCFD
body: Nice work @josephleekl! Will approve after `codecov` turns green. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T22:27:58Z
user: login: multiphaseCFD
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T13:09:44Z
user: login: josephleekl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T13:10:47Z
user: login: multiphaseCFD
body: :100: for `90x` speed-up
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-24T13:13:44Z
user: login: LuisAlfredoNu
body: Thank you @josephleekl 
Nice performance improment! 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T14:01:49Z

Review comments:
diff_hunk: @@ -384,3 +384,57 @@ def test_PauliZ_hadamard_PauliY(self, theta, phi, varphi, dev, tol):
         tol = 1e-5 if dev.c_dtype == np.complex64 else 1e-7
 
         assert np.allclose(calculated_val, reference_val, tol)
+
+    def test_permute_order(self, theta, phi, varphi, dev, tol):
path: tests/lightning_base/test_expval.py
user: login: josephleekl
body: LGPU MPI already has extensive tests for expval with hamiltonian with many terms
created_at: 2025-03-21T20:11:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -173,6 +173,33 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            # pylint: disable=protected-access
+            coeffs, pauli_sentence = QuantumScriptSerializer(
+                self._qubit_state.device_name, self.dtype == np.complex64, self._use_mpi
+            )._pauli_sentence(measurementprocess.obs.pauli_rep, direct_return=True)
+
+            wires = [[] for _ in range(len(coeffs))]
+            pauli_words = [None] * len(coeffs)
+            pauli_map = {
+                "PauliX": "X",
+                "PauliY": "Y",
+                "PauliZ": "Z",
+                "Identity": "I",
+            }
+
+            for i, pauli_word in enumerate(pauli_sentence):
+                ops = pauli_word.get_ops() if len(pauli_word.get_wires()) > 1 else [pauli_word]
+                pauli_word_wires = []
+                pauli_chars = []
+                for op in ops:
+                    pauli_word_wires.extend(op.get_wires())
+                    pauli_chars.append(pauli_map[op.get_base_ob_name()])
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: LuisAlfredoNu
body: You don't need to call a function from C++. Essentially, you are adding an intermediate step to define the Pauli character. 
From the serializer, you get the full name from C++ -> convert to a single character -> get the character from the serializer -> call C++ to convert the single character to the full name 
created_at: 2025-03-21T21:03:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -343,15 +343,18 @@ def map_wire(wire: int):
             ]
         )
 
-    def _pauli_sentence(self, observable, wires_map: dict = None):
+    def _pauli_sentence(self, observable, wires_map: dict = None, direct_return=False):
path: pennylane_lightning/core/_serialize.py
user: login: multiphaseCFD
body: I'm wondering if we would like to add docstr for the newly added args.
created_at: 2025-03-21T21:06:17Z
author_association: MEMBER
diff_hunk: @@ -173,6 +173,33 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            # pylint: disable=protected-access
+            coeffs, pauli_sentence = QuantumScriptSerializer(
+                self._qubit_state.device_name, self.dtype == np.complex64, self._use_mpi
+            )._pauli_sentence(measurementprocess.obs.pauli_rep, direct_return=True)
+
+            wires = [[] for _ in range(len(coeffs))]
+            pauli_words = [None] * len(coeffs)
+            pauli_map = {
+                "PauliX": "X",
+                "PauliY": "Y",
+                "PauliZ": "Z",
+                "Identity": "I",
+            }
+
+            for i, pauli_word in enumerate(pauli_sentence):
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: multiphaseCFD
body: Could we use [pauli_word_to_string](https://docs.pennylane.ai/en/stable/code/api/pennylane.pauli.pauli_word_to_string.html) to convert a Pauli word to a string?
created_at: 2025-03-21T21:15:38Z
author_association: MEMBER
diff_hunk: @@ -173,6 +173,33 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            # pylint: disable=protected-access
+            coeffs, pauli_sentence = QuantumScriptSerializer(
+                self._qubit_state.device_name, self.dtype == np.complex64, self._use_mpi
+            )._pauli_sentence(measurementprocess.obs.pauli_rep, direct_return=True)
+
+            wires = [[] for _ in range(len(coeffs))]
+            pauli_words = [None] * len(coeffs)
+            pauli_map = {
+                "PauliX": "X",
+                "PauliY": "Y",
+                "PauliZ": "Z",
+                "Identity": "I",
+            }
+
+            for i, pauli_word in enumerate(pauli_sentence):
+                ops = pauli_word.get_ops() if len(pauli_word.get_wires()) > 1 else [pauli_word]
+                pauli_word_wires = []
+                pauli_chars = []
+                for op in ops:
+                    pauli_word_wires.extend(op.get_wires())
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: multiphaseCFD
body: Could we use `PauliWords.wires` to get the wires directly?
created_at: 2025-03-21T21:20:31Z
author_association: MEMBER
diff_hunk: @@ -173,6 +173,33 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            # pylint: disable=protected-access
+            coeffs, pauli_sentence = QuantumScriptSerializer(
+                self._qubit_state.device_name, self.dtype == np.complex64, self._use_mpi
+            )._pauli_sentence(measurementprocess.obs.pauli_rep, direct_return=True)
+
+            wires = [[] for _ in range(len(coeffs))]
+            pauli_words = [None] * len(coeffs)
+            pauli_map = {
+                "PauliX": "X",
+                "PauliY": "Y",
+                "PauliZ": "Z",
+                "Identity": "I",
+            }
+
+            for i, pauli_word in enumerate(pauli_sentence):
+                ops = pauli_word.get_ops() if len(pauli_word.get_wires()) > 1 else [pauli_word]
+                pauli_word_wires = []
+                pauli_chars = []
+                for op in ops:
+                    pauli_word_wires.extend(op.get_wires())
+                    pauli_chars.append(pauli_map[op.get_base_ob_name()])
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: josephleekl
body: Good point, I'll update it!
created_at: 2025-03-21T21:21:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -343,15 +343,18 @@ def map_wire(wire: int):
             ]
         )
 
-    def _pauli_sentence(self, observable, wires_map: dict = None):
+    def _pauli_sentence(self, observable, wires_map: dict = None, direct_return=False):
path: pennylane_lightning/core/_serialize.py
user: login: josephleekl
body: no longer necessary!
created_at: 2025-03-21T22:23:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -173,6 +173,33 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            # pylint: disable=protected-access
+            coeffs, pauli_sentence = QuantumScriptSerializer(
+                self._qubit_state.device_name, self.dtype == np.complex64, self._use_mpi
+            )._pauli_sentence(measurementprocess.obs.pauli_rep, direct_return=True)
+
+            wires = [[] for _ in range(len(coeffs))]
+            pauli_words = [None] * len(coeffs)
+            pauli_map = {
+                "PauliX": "X",
+                "PauliY": "Y",
+                "PauliZ": "Z",
+                "Identity": "I",
+            }
+
+            for i, pauli_word in enumerate(pauli_sentence):
+                ops = pauli_word.get_ops() if len(pauli_word.get_wires()) > 1 else [pauli_word]
+                pauli_word_wires = []
+                pauli_chars = []
+                for op in ops:
+                    pauli_word_wires.extend(op.get_wires())
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: josephleekl
body: indeed, just wires.tolist!
created_at: 2025-03-21T22:23:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -173,6 +173,33 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            # pylint: disable=protected-access
+            coeffs, pauli_sentence = QuantumScriptSerializer(
+                self._qubit_state.device_name, self.dtype == np.complex64, self._use_mpi
+            )._pauli_sentence(measurementprocess.obs.pauli_rep, direct_return=True)
+
+            wires = [[] for _ in range(len(coeffs))]
+            pauli_words = [None] * len(coeffs)
+            pauli_map = {
+                "PauliX": "X",
+                "PauliY": "Y",
+                "PauliZ": "Z",
+                "Identity": "I",
+            }
+
+            for i, pauli_word in enumerate(pauli_sentence):
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: josephleekl
body: that's great, now it's MUCH cleaner...
created_at: 2025-03-21T22:23:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -173,6 +173,13 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            pwords, coeffs = zip(*measurementprocess.obs.pauli_rep.items())
+            pauli_words = [qml.pauli.pauli_word_to_string(p) for p in pwords]
+            wires = [p.wires.tolist() for p in pwords]
+            return self._measurement_lightning.expval(pauli_words, wires, coeffs)
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: multiphaseCFD
body: hmmm, the codecov still complains?
created_at: 2025-03-24T13:09:44Z
author_association: MEMBER
diff_hunk: @@ -173,6 +173,13 @@ def expval(self, measurementprocess: MeasurementProcess):
                 CSR_SparseHamiltonian.data,
             )
 
+        # use specialized function to compute expval(pauli_sentence)
+        if measurementprocess.obs.pauli_rep is not None:
+            pwords, coeffs = zip(*measurementprocess.obs.pauli_rep.items())
+            pauli_words = [qml.pauli.pauli_word_to_string(p) for p in pwords]
+            wires = [p.wires.tolist() for p in pwords]
+            return self._measurement_lightning.expval(pauli_words, wires, coeffs)
path: pennylane_lightning/lightning_gpu/_measurements.py
user: login: josephleekl
body: yeah, codecov GPU report is faulty. From codecov itself it says its covered https://app.codecov.io/gh/PennyLaneAI/pennylane-lightning/pull/1104?dropdown=coverage&src=pr&el=h1
created_at: 2025-03-24T13:10:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-03-20T22:19:39Z
body: ## [Codecov](https://app.codecov.io/gh/PennyLaneAI/pennylane-lightning/pull/1104?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PennyLaneAI) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 98.92%. Comparing base [(`888e226`)](https://app.codecov.io/gh/PennyLaneAI/pennylane-lightning/commit/888e226eccff9d5fdf287d57d1fda0fecab04784?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PennyLaneAI) to head [(`d47d363`)](https://app.codecov.io/gh/PennyLaneAI/pennylane-lightning/commit/d47d36307d6fc2f1426236f021d6cb13625a6a2a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PennyLaneAI).
> Report is 52 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #1104      +/-   ##
==========================================
+ Coverage   95.80%   98.92%   +3.12%     
==========================================
  Files         237      271      +34     
  Lines       40188    45641    +5453     
==========================================
+ Hits        38501    45152    +6651     
+ Misses       1687      489    -1198     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/PennyLaneAI/pennylane-lightning/pull/1104?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PennyLaneAI).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=PennyLaneAI).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE