Repository: googleapis/google-cloud-python

Issue:
Core: Race condition in bidi.BackgroundConsumer?
#### Environment details
Any OS, any Python version, google-api-core==1.9.0.


#### Steps to reproduce
No actual steps, spotted this in [bidi.py](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py) while working on a different issue. This is how the _"Am I paused?"_ [snippet](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L536-L540) in `BackgroundConsumer._thread_main()` looks like:
```py
with self._wake:
    if self._paused:
        _LOGGER.debug("paused, waiting for waking.")
        self._wake.wait()
        _LOGGER.debug("woken.")

_LOGGER.debug("waiting for recv.")
response = self._bidi_rpc.recv()
```
The `_paused` state can be set / cleared by the [pause() and resume()](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L605-L617) methods, respectively.

If paused, the code snippet from above blocks at the `_wake.wait()` call, and is unblocked some time after [_wake.notifyAll()](https://github.com/googleapis/google-cloud-python/blob/master/api_core/google/api_core/bidi.py#L617) is invoked in the `resume()` method. When `resume()` method notifies the waiting threads and releases the internal lock held by the `self._wake` condition, `_wake.wait()` tries to re-obtain that lock.

Now, suppose that some other thread invokes the `pause()` method in the meantime, and the latter obtains the `self._wake`'s lock before `_wake.wait()` can grab it. The `_paused` flag will again be set to `True`, and when `_wake.wait()` finally acquires the lock and resumes, the code will invoke `_bidi_rpc.recv()` in the _paused_ state.

For this reason the `self._paused` condition must be checked in a loop, and not in a single `if` statement, as the docs on [threading.Condition](https://docs.python.org/3.7/library/threading.html#condition-objects) state:

> The while loop checking for the application’s condition is necessary because `wait()` can return after an arbitrary long time, and the condition which prompted the `notify()` call may no longer hold true. This is inherent to multi-threaded programming.

How much of a problem this is, i.e. invoking `recv()` in a paused state?

---
**Edit:**
The same can happen when exiting the `with self._wake` block. The lock gets released, `pause()` can acquire it and set `_self._paused` to `True`, but `_bidi_rpc.recv()` will nevertheless be called, because it is placed outside the `with` block.

---

#### Code example

I created a [demo script](https://gist.github.com/plamut/b8937bdf2f6663e3e9fc503c5ccf1983) that demonstrates why `if <condition>` is not sufficient, and subject to race conditions (requires Python 3.6+). The "worker" thread waits until a shared global number is set to 10, while five "changer threads" randomly change that number, and notify other threads if they change it to 10. 

Sometimes the "worker" is invoked too late, and another "changer" thread changes 10 to something else, causing the "worker" to continue running when the target condition is no longer true. Replacing `if not <condition>` with `while not <condition>` gets rid of the bug.

```
...
INFO     [2019-04-29 23:09:20,327] Thread-Changer-0: Changing number from  9 to 10
INFO     [2019-04-29 23:09:20,327] Thread-Changer-0: New number == 10, notifying others
INFO     [2019-04-29 23:09:20,327] Thread-Worker   : I see the number 10
INFO     [2019-04-29 23:09:20,327] Thread-Changer-1: Changing number from 10 to  4
INFO     [2019-04-29 23:09:20,327] Thread-Changer-4: Changing number from  4 to 10
INFO     [2019-04-29 23:09:20,327] Thread-Changer-4: New number == 10, notifying others
INFO     [2019-04-29 23:09:20,328] Thread-Changer-2: Changing number from 10 to  3
INFO     [2019-04-29 23:09:21,328] Thread-Changer-3: Changing number from  3 to  9
INFO     [2019-04-29 23:09:21,328] Thread-Changer-1: Changing number from  9 to  6
INFO     [2019-04-29 23:09:21,328] Thread-Changer-4: Changing number from  6 to  4
INFO     [2019-04-29 23:09:21,328] Thread-Worker   : the number is not 10, will wait for condition
INFO     [2019-04-29 23:09:21,328] Thread-Changer-0: Changing number from  4 to  6
INFO     [2019-04-29 23:09:21,329] Thread-Changer-2: Changing number from  6 to  9
INFO     [2019-04-29 23:09:22,329] Thread-Changer-3: Changing number from  9 to 10
INFO     [2019-04-29 23:09:22,329] Thread-Changer-3: New number == 10, notifying others
INFO     [2019-04-29 23:09:22,329] Thread-Changer-1: Changing number from 10 to  5
INFO     [2019-04-29 23:09:22,329] Thread-Worker   : I see the number 5
...
```

The expected behavior is that the worker thread _always_ prints out _"I see the number 10"_ (the desired condition), and never _"I see <non 10>"_. In other words, it should only proceed when the condition `number == 10` is currently fulfilled.

In the `bidi.py` case, this would translate to `BackgroundConsumer._thread_main()` only resuming when `self._paused == False`, and _never_ resuming when `self._paused == True`.


Discussion:
Under the cover, you can't really delete a property, at best you can replace the content of an entity with the same minus the property you want to remove.

Are you looking forward hiding this form the user? This could give the wrong idea that this is a cheap operation.

That's exactly what i'd like to document:

``` python
del entity['prop_to_delete']
entity.save()
```

We should be clear that a save() is a save() -- no special "delete this single property" operation.

@jgeewax SGTM

PR #258 addresses this while fixing #167.


Resolving patch:
diff --git a/api_core/google/api_core/bidi.py b/api_core/google/api_core/bidi.py
--- a/api_core/google/api_core/bidi.py
+++ b/api_core/google/api_core/bidi.py
@@ -642,7 +642,7 @@ def _thread_main(self, ready):
                 # In the future, we could use `Condition.wait_for` if we drop
                 # Python 2.7.
                 with self._wake:
-                    if self._paused:
+                    while self._paused:
                         _LOGGER.debug("paused, waiting for waking.")
                         self._wake.wait()
                         _LOGGER.debug("woken.")


Test patch: