Repository: PennyLaneAI/pennylane

Issue:
Warn/raise in case shot noise and caching are enabled at the same time?
### Feature details

When `default.qubit` with `shots != None` but also `cache=1` is used then the same (noisy) result is returned again and again when a `QNode` is called. It is hard to imagine a case where this is what a user wants. 

Either they want a `QNode` to return a noisy result that is different each time it is called (they can get this by setting `cache=0`), or they want only a single noisy result, in which case caching doesn't provide any benefit.

I think it would be nice to raise an exception or at least a warning when a user makes a device with shot noise and caching enabled.

Alternatively the the device could go back to state caching, as it used to be in ancient times... This would give a speed benefit because the circuit would not need to be re-evaluated when the same shot-noisy `QNode` is called several times.
    

### Implementation

To demonstrate the "problem" consider this sample program:
```
import pennylane as qml
from pennylane import numpy as np

shots=76

dev = qml.device(
    'default.qubit',
    wires=2,
    shots=shots,
    cache=1,
)

np.random.seed(87)
params = np.random.randn(2, 2, 3)
observable_as_array = np.array([[ 1. +0.j,  0. +0.j,  0. +0.j,  0. +0.j],
                                [ 0. +0.j,  0.4+0.j, 21.6+0.j,  0. +0.j],
                                [ 0. +0.j, 21.6+0.j, -0.4+0.j,  0. +0.j],
                                [ 0. +0.j,  0. +0.j,  0. +0.j, -1. +0.j]])

@qml.qnode(dev, diff_method='parameter-shift')
def circuit(params):
    qml.templates.layers.StronglyEntanglingLayers(params, wires=range(dev.num_wires))
    return qml.expval(qml.Hermitian(observable_as_array, wires=range(dev.num_wires)))

for _ in range(10):
    print(circuit(params))
```

### How important would you say this feature is?

1: Not important. Would be nice to have.

### Additional information

_No response_


Discussion:
Thanks for reporting this @cvjjm!

I'm curious if this is fixed in the new caching system that is provided by the QNode. 
This new caching system is designed to be device independent, and allows creating your own cache object (it can be simply a dictionary, or a `cachetools.LRUCache`, for example), and pass it to QNodes you want to use it in:

```python
dev = qml.device('default.qubit', wires=nwires)

cache = {}

@qml.qnode(dev, diff_method='parameter-shift', cache=cache)
def expval_circuit(params):
    qml.templates.BasicEntanglerLayers(params, wires=wires, rotation=qml.RX)
    return qml.expval(qml.Hermitian(hamiltonian, wires=wires))

@qml.qnode(dev, diff_method='parameter-shift', cache=cache)
def var_circuit(params):
    qml.templates.BasicEntanglerLayers(params, wires=wires, rotation=qml.RX)
    return qml.var(qml.Hermitian(hamiltonian, wires=wires))

>>> print(expval_circuit(params), var_circuit(params)))
-2.6224875809703287, 0.011605015096940896
>>> print(cache)
{-2104543859499562169: array([-2.62248758]), 3268478774057874589: array([0.01160502])}
>>> print(expval_circuit(params), var_circuit(params)))  # evaluate QNodes again
-2.6224875809703287, 0.011605015096940896
>>> dev.num_executions  # only the first two executions will register
2
```

The plan is to make this the new user-facing caching system, and deprecate/remove the device-level one.

This is a very nice UI. Caching on Qnode level is a very good idea now that results are cached ad not states (for the latter it used to be natural to do it on a device level). 

The new cache shows the same behavior.
```
import pennylane as qml
from pennylane import numpy as np

shots=76

dev = qml.device(
    'default.qubit',
    wires=2,
    shots=shots,
    cache=0,
)

np.random.seed(87)
params = np.random.randn(2, 2, 3)
observable_as_array = np.array([[ 1. +0.j,  0. +0.j,  0. +0.j,  0. +0.j],
                                [ 0. +0.j,  0.4+0.j, 21.6+0.j,  0. +0.j],
                                [ 0. +0.j, 21.6+0.j, -0.4+0.j,  0. +0.j],
                                [ 0. +0.j,  0. +0.j,  0. +0.j, -1. +0.j]])

cache = {}

@qml.qnode(dev, diff_method='parameter-shift', cache=cache)
def circuit(params):
    qml.templates.layers.StronglyEntanglingLayers(params, wires=range(dev.num_wires))
    return qml.expval(qml.Hermitian(observable_as_array, wires=range(dev.num_wires)))

for _ in range(10):
    print(circuit(params))

print(cache)
```
Output:
```
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
6.403654883035922
{8257838482108885402: array([6.40365488])}
```
 This is not a bid deal but I think it can surprise users and disallowing the combination of shot noise and caching or raising a warning could be nice.
> This is not a bid deal but I think it can surprise users and disallowing the combination of shot noise and caching or raising a warning could be nice.

I agree, this is a good idea!
This is mostly resolved, the only thing that remains is adding the warning when shot noise and caching are combined.
@cvjjm thanks for the reminder!
Hi @cvjjm,
> This is mostly resolved, the only thing that remains is adding the warning when shot noise and caching are combined.

just to double-check: the `QubitDevice` caching capability is to be removed (a deprecation warning is now emitted).

The warning should be added when caching on Qnode level, correct?
Yes, that is entirely correct. 
Not sure if this is related, but @anthayes92 and myself have gotten surprised by the following using `qml.execute()`. Suppose we have a list that contains the same tape more than once:
```python
import pennylane as qml

dev = qml.device("default.qubit", wires=1, shots=3)

with qml.tape.QuantumTape() as tape:
    qml.RX(1.5, wires=0)
    qml.sample(wires=0)

tapes = [tape, tape]
```
Running `qml.execute()` gives:
```
>>> qml.execute(tapes, dev, gradient_fn=None)
[array([[[1],
         [0],
         [0]]]),
 array([[[1],
         [0],
         [0]]])]
```
i.e., we get the same set of samples each time. To get what we expect, we have to do:
```
>>> qml.execute(tapes, dev, gradient_fn=None, cache=False)
[array([[[1],
         [0],
         [1]]]),
 array([[[1],
         [0],
         [0]]])]
```
Perhaps this can be fixed with the warning mentioned above, i.e., when using caching in finite shots mode.
@trbromley good catch -- this is not a case I considered (submitting the same tape multiple times in a single call) when I coded up `qml.execute()`.

I'm curious to the reason behind this usage, could you instead do `qml.execute([tape], ...)`, but specify a batch of shots?
Thanks @josh146, yes it's maybe a bit of a fringe usecase.
Suppose we have a function/transform that takes an input circuit and randomly alters it (e.g., performing some random state preps/measurements), and outputs a sequence of such randomly altered tapes. We then want to execute these tapes, each with a fixed number of shots. We can call `qml.execute(tapes, ...)`, but there is a chance that our random generation has created a "collision" and we'll have the same circuit being run twice, resulting in repeated samples outlined above. An alternative could be to count how many times a given circuit has been randomly generated and alter the shots accordingly, but I don't think `qml.execute` supports a differerent shot value for each input tape?
Good points @trbromley, this is a good reason to support the same tape multiple times 🤔 

>  but I don't think qml.execute supports a differerent shot value for each input tape?

It doesn't atm, but this is something I would like to add - allow `shots` to be metadata on tape objects.

Resolving patch:
diff --git a/pennylane/interfaces/execution.py b/pennylane/interfaces/execution.py
--- a/pennylane/interfaces/execution.py
+++ b/pennylane/interfaces/execution.py
@@ -21,6 +21,9 @@
 # pylint: disable=import-outside-toplevel,too-many-arguments,too-many-branches,not-callable
 from functools import wraps
 import itertools
+import warnings
+import inspect
+from contextlib import _GeneratorContextManager
 from cachetools import LRUCache
 
 import pennylane as qml
@@ -119,6 +122,33 @@ def wrapper(tapes, **kwargs):
             if hashes[i] in cache:
                 # Tape exists within the cache, store the cached result
                 cached_results[i] = cache[hashes[i]]
+
+                # Introspect the set_shots decorator of the input function:
+                #   warn the user in case of finite shots with cached results
+                finite_shots = False
+
+                closure = inspect.getclosurevars(fn).nonlocals
+                if "original_fn" in closure:  # deal with expand_fn wrapper above
+                    closure = inspect.getclosurevars(closure["original_fn"]).nonlocals
+
+                # retrieve the captured context manager instance (for set_shots)
+                if "self" in closure and isinstance(closure["self"], _GeneratorContextManager):
+                    # retrieve the shots from the arguments or device instance
+                    if closure["self"].func.__name__ == "set_shots":
+                        dev, shots = closure["self"].args
+                        shots = dev.shots if shots is False else shots
+                        finite_shots = isinstance(shots, int)
+
+                if finite_shots:
+                    warnings.warn(
+                        "Cached execution with finite shots detected!\n"
+                        "Note that samples as well as all noisy quantities computed via sampling "
+                        "will be identical across executions. This situation arises where tapes "
+                        "are executed with identical operations, measurements, and parameters.\n"
+                        "To avoid this behavior, provide 'cache=False' to the QNode or execution "
+                        "function.",
+                        UserWarning,
+                    )
             else:
                 # Tape does not exist within the cache, store the tape
                 # for execution via the execution function.


Test patch:
diff --git a/tests/test_qnode.py b/tests/test_qnode.py
--- a/tests/test_qnode.py
+++ b/tests/test_qnode.py
@@ -14,6 +14,7 @@
 """Unit tests for the QNode"""
 from collections import defaultdict
 import pytest
+import warnings
 import numpy as np
 from scipy.sparse import coo_matrix
 
@@ -1092,6 +1093,62 @@ def circuit(a):
         assert len(res) == 2
         assert dev.shots == 3
 
+    def test_warning_finite_shots_dev(self):
+        """Tests that a warning is raised when caching is used with finite shots."""
+        dev = qml.device("default.qubit", wires=1, shots=5)
+
+        @qml.qnode(dev, cache={})
+        def circuit(x):
+            qml.RZ(x, wires=0)
+            return qml.expval(qml.PauliZ(0))
+
+        # no warning on the first execution
+        circuit(0.3)
+        with pytest.warns(UserWarning, match="Cached execution with finite shots detected"):
+            circuit(0.3)
+
+    def test_warning_finite_shots_override(self):
+        """Tests that a warning is raised when caching is used with finite shots."""
+        dev = qml.device("default.qubit", wires=1)
+
+        @qml.qnode(dev, cache={})
+        def circuit(x):
+            qml.RZ(x, wires=0)
+            return qml.expval(qml.PauliZ(0))
+
+        # no warning on the first execution
+        circuit(0.3)
+        with pytest.warns(UserWarning, match="Cached execution with finite shots detected"):
+            circuit(0.3, shots=5)
+
+    def test_warning_finite_shots_tape(self):
+        """Tests that a warning is raised when caching is used with finite shots."""
+        dev = qml.device("default.qubit", wires=1, shots=5)
+
+        with qml.tape.QuantumTape() as tape:
+            qml.RZ(0.3, wires=0)
+            qml.expval(qml.PauliZ(0))
+
+        # no warning on the first execution
+        cache = {}
+        qml.execute([tape], dev, None, cache=cache)
+        with pytest.warns(UserWarning, match="Cached execution with finite shots detected"):
+            qml.execute([tape], dev, None, cache=cache)
+
+    def test_no_warning_infinite_shots(self):
+        """Tests that no warning is raised when caching is used with infinite shots."""
+        dev = qml.device("default.qubit", wires=1)
+
+        @qml.qnode(dev, cache={})
+        def circuit(x):
+            qml.RZ(x, wires=0)
+            return qml.expval(qml.PauliZ(0))
+
+        with warnings.catch_warnings():
+            warnings.filterwarnings("error", message="Cached execution with finite shots detected")
+            circuit(0.3)
+            circuit(0.3)
+
 
 @pytest.mark.xfail
 class TestSpecs: