Repository: Qiskit/qiskit

Issue:
Infinite job submissions when submitted jobs' status is error/cancelled/unknown status
<!-- ⚠️ If you do not respect this template, your issue will be closed -->
<!-- ⚠️ Make sure to browse the opened and closed issues -->

### Information

- **Qiskit Terra version**: lastest
- **Python version**:
- **Operating system**:

### What is the current behavior?
The problem is in the run_qobj() function in qiskit/utils/run_circuits.py.
When a submitted job to a non-simulator backend is in a JobStatus.ERROR or JobStatus.CANCELLED status, the job is submitted again. This happens infinitely many times if the job is continually canceled or in an error state.           
```python
if job_status == JobStatus.CANCELLED:
    logger.warning("FAILURE: Job id: %s is cancelled. Re-submit the Qobj.", job_id)
elif job_status == JobStatus.ERROR:
    logger.warning(
        "FAILURE: Job id: %s encounters the error. "
        "Error is : %s. Re-submit the Qobj.",
        job_id,
        job.error_message(),
    )
else:
    logging.warning(
        "FAILURE: Job id: %s. Unknown status: %s. " "Re-submit the Qobj.",
        job_id,
        job_status,
    )
job, job_id = _safe_submit_qobj(
    qobj, backend, backend_options, noise_config, skip_qobj_validation
)

```


This is especially dangerous when a the job was submitted successfully, but reading the status was wrong (due to a bug in the job class or at the backend side), possibly causing infinite job submissions each costing the user money.

### Steps to reproduce the problem
Execute a job that always returns the status JobStatus.CANCELLED.

### What is the expected behavior?
The job submission should stop after a finite amount of retries.

### Suggested solutions
Do not use `while True` loops in the code. Instead use a finite, preferably small, amount of retries.


Quantum instance job submission has infinite loop
<!-- ⚠️ If you do not respect this template, your issue will be closed -->
<!-- ⚠️ Make sure to browse the opened and closed issues -->

### Information

- **Qiskit Terra version**: latest
- **Python version**:
- **Operating system**:

### What is the current behavior?
Submitting jobs via the `quantum_instance` has an infinite loop that does not exit unless the job was submitted correctly and a `job_id` can be grabbed: 

https://github.com/Qiskit/qiskit-terra/blob/0c6890dc0ed55626e96ceb5c27234ed2116e76bb/qiskit/utils/run_circuits.py#L149

If this does not happen, the code loops forever emitting warnings.

Reference: https://github.com/Qiskit-Partners/qiskit-aqt-provider/issues/46

### Steps to reproduce the problem



### What is the expected behavior?
Qiskit does not try infinitely many times to submit jobs.


### Suggested solutions
No `while True:` loops.




Discussion:
This is related to #6821 but not the same bug. #6821 is an infinite loop of trying to create a job and failing, while this is an infinite loop of submitting the job. This is even more dangerous due to the possibility of infinite job submissions, each costing the user.
I think the best practice would be to eliminate all `while True` loops and replace with a finite (preferably small) number of retries, and fail gracefully otherwise.
@shahakL may I look at this issue ?
Also, it would be quite helpful if you could tell me how I can reproduce this bug with code. Thanks!

@TheGupta2012 Sure!
Please see the attached piece of code for reference. It should call the MyBackend.run() function an infinite amount of times, therefore creating infinite jobs.
In addition, like seen in #6821, there are many additional scenarios for infinite loops when submitting jobs through a QuantumInstance (e.g. if a job constantly returns a CANCELLED status, it enters an infinite loop). So I believe that all of the "while True" loops inside the run_circuits.py file should be fixed.

[main.py.zip](https://github.com/Qiskit/qiskit-terra/files/6973576/main.py.zip)

Thanks @shahakL ! I'll update you when I make some progress.

@shahakL one question I had was that if we **max** out on the **number of trials** (let's say 10 instead of infinite) for one of the jobs in a particular job set, do we raise an `Exception` and make the user submit the **whole job set again** or just skip that job and try to continue ahead, in that job set ?
@nonhermitian could you provide some example of how I may reproduce this in code? 

Is this only on jobs to `IBMQBackend` or is this also to Aer?
Please see the attached code for an example
[main.py.zip](https://github.com/Qiskit/qiskit-terra/files/6973623/main.py.zip)


Resolving patch:
diff --git a/qiskit/utils/quantum_instance.py b/qiskit/utils/quantum_instance.py
--- a/qiskit/utils/quantum_instance.py
+++ b/qiskit/utils/quantum_instance.py
@@ -136,6 +136,7 @@ def __init__(
         measurement_error_mitigation_shots: Optional[int] = None,
         job_callback: Optional[Callable] = None,
         mit_pattern: Optional[List[List[int]]] = None,
+        max_job_retries: int = 50,
     ) -> None:
         """
         Quantum Instance holds a Qiskit Terra backend as well as configuration for circuit
@@ -184,6 +185,8 @@ def __init__(
                 measurement correction, divided to groups according to tensors.
                 If `None` and `qr` is given then assumed to be performed over the entire
                 `qr` as one group (default `None`).
+            max_job_retries(int): positive non-zero number of trials for the job set (-1 for
+                infinite trials) (default: 50)
 
         Raises:
             QiskitError: the shots exceeds the maximum number of shots
@@ -302,6 +305,7 @@ def __init__(
         self._circuit_summary = False
         self._job_callback = job_callback
         self._time_taken = 0.0
+        self._max_job_retries = max_job_retries
         logger.info(self)
 
     def __str__(self) -> str:
@@ -323,6 +327,7 @@ def __str__(self) -> str:
             self._backend_options,
             self._noise_config,
         )
+
         info += f"\nMeasurement mitigation: {self._meas_error_mitigation_cls}"
 
         return info
@@ -378,6 +383,7 @@ def execute(self, circuits, had_transpiled: bool = False):
             QiskitError: TensoredMeasFitter class doesn't support subset fitter
             MissingOptionalLibraryError: Ignis not installed
 
+
         Returns:
             Result: result object
 
@@ -385,7 +391,6 @@ def execute(self, circuits, had_transpiled: bool = False):
               assembling to the qobj.
         """
         from qiskit.utils.run_circuits import run_qobj, run_circuits
-
         from qiskit.utils.measurement_error_mitigation import (
             get_measured_qubits_from_qobj,
             get_measured_qubits,
@@ -502,6 +507,7 @@ def _find_save_state(data):
                             noise_config=self._noise_config,
                             run_config=self._run_config.to_dict(),
                             job_callback=self._job_callback,
+                            max_job_retries=self._max_job_retries,
                         )
                         self._time_taken += cals_result.time_taken
                         result = run_circuits(
@@ -512,6 +518,7 @@ def _find_save_state(data):
                             noise_config=self._noise_config,
                             run_config=self.run_config.to_dict(),
                             job_callback=self._job_callback,
+                            max_job_retries=self._max_job_retries,
                         )
                         self._time_taken += result.time_taken
                     else:
@@ -524,6 +531,7 @@ def _find_save_state(data):
                             noise_config=self._noise_config,
                             run_config=self.run_config.to_dict(),
                             job_callback=self._job_callback,
+                            max_job_retries=self._max_job_retries,
                         )
                         self._time_taken += result.time_taken
                         cals_result = result
@@ -560,6 +568,7 @@ def _find_save_state(data):
                             self._noise_config,
                             self._skip_qobj_validation,
                             self._job_callback,
+                            self._max_job_retries,
                         )
                         self._time_taken += cals_result.time_taken
                         result = run_qobj(
@@ -570,6 +579,7 @@ def _find_save_state(data):
                             self._noise_config,
                             self._skip_qobj_validation,
                             self._job_callback,
+                            self._max_job_retries,
                         )
                         self._time_taken += result.time_taken
                     else:
@@ -583,6 +593,7 @@ def _find_save_state(data):
                             self._noise_config,
                             self._skip_qobj_validation,
                             self._job_callback,
+                            self._max_job_retries,
                         )
                         self._time_taken += result.time_taken
                         cals_result = result
@@ -611,6 +622,7 @@ def _find_save_state(data):
                         noise_config=self._noise_config,
                         run_config=self._run_config.to_dict(),
                         job_callback=self._job_callback,
+                        max_job_retries=self._max_job_retries,
                     )
                     if circuit_job
                     else run_qobj(
@@ -621,6 +633,7 @@ def _find_save_state(data):
                         self._noise_config,
                         self._skip_qobj_validation,
                         self._job_callback,
+                        self._max_job_retries,
                     )
                 )
                 self._time_taken += result.time_taken
@@ -674,6 +687,7 @@ def _find_save_state(data):
                     noise_config=self._noise_config,
                     run_config=self._run_config.to_dict(),
                     job_callback=self._job_callback,
+                    max_job_retries=self._max_job_retries,
                 )
                 if circuit_job
                 else run_qobj(
@@ -684,6 +698,7 @@ def _find_save_state(data):
                     self._noise_config,
                     self._skip_qobj_validation,
                     self._job_callback,
+                    self._max_job_retries,
                 )
             )
             self._time_taken += result.time_taken
@@ -782,6 +797,25 @@ def circuit_summary(self, new_value):
         """sets circuit summary"""
         self._circuit_summary = new_value
 
+    @property
+    def max_job_retries(self):
+        """Getter of max tries"""
+        return self._max_job_retries
+
+    @max_job_retries.setter
+    def max_job_retries(self, new_value):
+        """Sets the maximum tries"""
+        if not isinstance(new_value, int):
+            raise TypeError("max_job_retries parameter must be an integer")
+        if new_value < -1 or new_value == 0:
+            raise ValueError(
+                "max_job_retries must either be a positive integer or -1(for infinite trials)"
+            )
+        if new_value == -1:
+            self._max_job_retries = int(1e18)
+        else:
+            self._max_job_retries = new_value
+
     @property
     def measurement_error_mitigation_cls(self):  # pylint: disable=invalid-name
         """returns measurement error mitigation cls"""
diff --git a/qiskit/utils/run_circuits.py b/qiskit/utils/run_circuits.py
--- a/qiskit/utils/run_circuits.py
+++ b/qiskit/utils/run_circuits.py
@@ -144,9 +144,10 @@ def _safe_submit_qobj(
     backend_options: Dict,
     noise_config: Dict,
     skip_qobj_validation: bool,
+    max_job_retries: int,
 ) -> Tuple[BaseJob, str]:
     # assure get job ids
-    while True:
+    for _ in range(max_job_retries):
         try:
             job = run_on_backend(
                 backend,
@@ -185,7 +186,7 @@ def _safe_submit_qobj(
                             oldest_running.wait_for_final_state(timeout=300)
                         except Exception:  # pylint: disable=broad-except
                             # If the wait somehow fails or times out, we'll just re-try
-                            # the job submit and see if it works now.
+                            # the job submit and see if it works now ( but a finite number of times )
                             pass
             if failure_warn:
                 logger.warning(
@@ -197,13 +198,14 @@ def _safe_submit_qobj(
             logger.warning(
                 "FAILURE: Can not get job id, Resubmit the qobj to get job id. Error: %s ", ex
             )
+    else:
+        raise QiskitError("Max retry limit reached. Could not submit the qobj correctly")
 
     return job, job_id
 
 
-def _safe_get_job_status(job: BaseJob, job_id: str) -> JobStatus:
-
-    while True:
+def _safe_get_job_status(job: BaseJob, job_id: str, max_job_retries: int) -> JobStatus:
+    for _ in range(max_job_retries):
         try:
             job_status = job.status()
             break
@@ -216,10 +218,11 @@ def _safe_get_job_status(job: BaseJob, job_id: str) -> JobStatus:
             time.sleep(5)
         except Exception as ex:
             raise QiskitError(
-                "FAILURE: job id: {}, "
-                "status: 'FAIL_TO_GET_STATUS' "
-                "Unknown error: ({})".format(job_id, ex)
+                f"job id: {job_id}, status: 'FAIL_TO_GET_STATUS' Unknown error: ({ex})"
             ) from ex
+    else:
+        raise QiskitError(f"Max retry limit reached. Failed to get status for job with id {job_id}")
+
     return job_status
 
 
@@ -231,6 +234,7 @@ def run_qobj(
     noise_config: Optional[Dict] = None,
     skip_qobj_validation: bool = False,
     job_callback: Optional[Callable] = None,
+    max_job_retries: int = 50,
 ) -> Result:
     """
     An execution wrapper with Qiskit-Terra, with job auto recover capability.
@@ -244,11 +248,12 @@ def run_qobj(
         qjob_config: configuration for quantum job object
         backend_options: configuration for simulator
         noise_config: configuration for noise model
-        skip_qobj_validation: Bypass Qobj validation to decrease submission time,
-                                               only works for Aer and BasicAer providers
-        job_callback: callback used in querying info of the submitted job, and
-                                           providing the following arguments:
-                                            job_id, job_status, queue_position, job
+        skip_qobj_validation: Bypass Qobj validation to decrease submission time, only works for Aer
+            and BasicAer providers
+        job_callback: callback used in querying info of the submitted job, and providing the
+            following arguments: job_id, job_status, queue_position, job
+        max_job_retries(int): positive non-zero number of trials for the job set (-1 for infinite
+            trials) (default: 50).
 
     Returns:
         Result object
@@ -281,7 +286,7 @@ def run_qobj(
     job_ids = []
     for qob in qobjs:
         job, job_id = _safe_submit_qobj(
-            qob, backend, backend_options, noise_config, skip_qobj_validation
+            qob, backend, backend_options, noise_config, skip_qobj_validation, max_job_retries
         )
         job_ids.append(job_id)
         jobs.append(job)
@@ -294,11 +299,11 @@ def run_qobj(
         for idx, _ in enumerate(jobs):
             job = jobs[idx]
             job_id = job_ids[idx]
-            while True:
+            for _ in range(max_job_retries):
                 logger.info("Running %s-th qobj, job id: %s", idx, job_id)
                 # try to get result if possible
                 while True:
-                    job_status = _safe_get_job_status(job, job_id)
+                    job_status = _safe_get_job_status(job, job_id, max_job_retries)
                     queue_position = 0
                     if job_status in JOB_FINAL_STATES:
                         # do callback again after the job is in the final states
@@ -316,22 +321,25 @@ def run_qobj(
 
                 # get result after the status is DONE
                 if job_status == JobStatus.DONE:
-                    while True:
+                    for _ in range(max_job_retries):
                         result = job.result(**qjob_config)
                         if result.success:
                             results.append(result)
                             logger.info("COMPLETED the %s-th qobj, job id: %s", idx, job_id)
                             break
-
                         logger.warning("FAILURE: Job id: %s", job_id)
                         logger.warning(
                             "Job (%s) is completed anyway, retrieve result from backend again.",
                             job_id,
                         )
                         job = backend.retrieve_job(job_id)
+                    else:
+                        raise QiskitError(
+                            f"Max retry limit reached. Failed to get result for job id {job_id}"
+                        )
                     break
                 # for other cases, resubmit the qobj until the result is available.
-                # since if there is no result returned, there is no way algorithm can do any process
+                # since if there is no result returned, there is no way algorithm can do any processing
                 # get back the qobj first to avoid for job is consumed
                 qobj = job.qobj()
                 if job_status == JobStatus.CANCELLED:
@@ -349,12 +357,20 @@ def run_qobj(
                         job_id,
                         job_status,
                     )
-
                 job, job_id = _safe_submit_qobj(
-                    qobj, backend, backend_options, noise_config, skip_qobj_validation
+                    qobj,
+                    backend,
+                    backend_options,
+                    noise_config,
+                    skip_qobj_validation,
+                    max_job_retries,
                 )
                 jobs[idx] = job
                 job_ids[idx] = job_id
+            else:
+                raise QiskitError(
+                    f"Max retry limit reached. Failed to get result for job with id {job_id}."
+                )
     else:
         results = []
         for job in jobs:
@@ -428,6 +444,7 @@ def run_circuits(
     noise_config: Optional[Dict] = None,
     run_config: Optional[Dict] = None,
     job_callback: Optional[Callable] = None,
+    max_job_retries: int = 50,
 ) -> Result:
     """
     An execution wrapper with Qiskit-Terra, with job auto recover capability.
@@ -442,9 +459,10 @@ def run_circuits(
         backend_options: backend options
         noise_config: configuration for noise model
         run_config: configuration for run
-        job_callback: callback used in querying info of the submitted job, and
-                                           providing the following arguments:
-                                            job_id, job_status, queue_position, job
+        job_callback: callback used in querying info of the submitted job, and providing the
+            following arguments: job_id, job_status, queue_position, job.
+        max_job_retries(int): positive non-zero number of trials for the job set (-1 for infinite
+            trials) (default: 50)
 
     Returns:
         Result object
@@ -480,6 +498,7 @@ def run_circuits(
                 backend_options=backend_options,
                 noise_config=noise_config,
                 run_config=run_config,
+                max_job_retries=max_job_retries,
             )
             jobs.append(job)
             job_ids.append(job_id)
@@ -492,6 +511,7 @@ def run_circuits(
             backend_options=backend_options,
             noise_config=noise_config,
             run_config=run_config,
+            max_job_retries=max_job_retries,
         )
         jobs = [job]
         job_ids = [job_id]
@@ -507,11 +527,13 @@ def run_circuits(
             logger.info("There is one jobs are submitted: id: %s", job_id)
             job = jobs[idx]
             job_id = job_ids[idx]
-            while True:
+            for _ in range(max_job_retries):
                 logger.info("Running job id: %s", job_id)
                 # try to get result if possible
                 while True:
-                    job_status = _safe_get_job_status(job, job_id)
+                    job_status = _safe_get_job_status(
+                        job, job_id, max_job_retries
+                    )  # if the status was broken, an Exception would be raised anyway
                     queue_position = 0
                     if job_status in JOB_FINAL_STATES:
                         # do callback again after the job is in the final states
@@ -529,7 +551,7 @@ def run_circuits(
 
                 # get result after the status is DONE
                 if job_status == JobStatus.DONE:
-                    while True:
+                    for _ in range(max_job_retries):
                         result = job.result()
                         if result.success:
                             results.append(result)
@@ -542,6 +564,10 @@ def run_circuits(
                             job_id,
                         )
                         job = backend.retrieve_job(job_id)
+                    else:
+                        raise QiskitError(
+                            f"Max retry limit reached. Failed to get result for job id {job_id}"
+                        )
                     break
                 # for other cases, resubmit the circuit until the result is available.
                 # since if there is no result returned, there is no way algorithm can do any process
@@ -562,7 +588,6 @@ def run_circuits(
                         job_id,
                         job_status,
                     )
-
                 job, job_id = _safe_submit_circuits(
                     split_circuits[idx],
                     backend,
@@ -570,6 +595,11 @@ def run_circuits(
                     backend_options=backend_options,
                     noise_config=noise_config,
                     run_config=run_config,
+                    max_job_retries=max_job_retries,
+                )
+            else:
+                raise QiskitError(
+                    f"Max retry limit reached. Failed to get result for job with id {job_id} "
                 )
     else:
         results = []
@@ -577,7 +607,6 @@ def run_circuits(
             results.append(job.result())
 
     result = _combine_result_objects(results) if results else None
-
     # If result was not successful then raise an exception with either the status msg or
     # extra information if this was an Aer partial result return
     if not result.success:
@@ -605,9 +634,10 @@ def _safe_submit_circuits(
     backend_options: Dict,
     noise_config: Dict,
     run_config: Dict,
+    max_job_retries: int,
 ) -> Tuple[BaseJob, str]:
     # assure get job ids
-    while True:
+    for _ in range(max_job_retries):
         try:
             job = _run_circuits_on_backend(
                 backend,
@@ -660,6 +690,8 @@ def _safe_submit_circuits(
             logger.warning(
                 "FAILURE: Can not get job id, Resubmit the qobj to get job id. Error: %s ", ex
             )
+    else:
+        raise QiskitError("Max retry limit reached. Failed to submit the qobj correctly")
 
     return job, job_id
 


Test patch: