Repository: open-telemetry/opentelemetry-python

Issue:
BatchExportSpanProcessor can degenerate into timeout-less busy-waiting
**Describe your environment** 
* Python version 3.7.9
* Opentelemetry version 0.12b0 (although bug is present in `master`)

**Steps to reproduce**
Hard to precisely simulate. Need to force a high-latency span export and then stop emitting spans. I can try and emulate with a code snippet later.

**What is the expected behavior?**
Minimal CPU usage from worker thread, especially if no spans are being processed.

**What is the actual behavior?**
`BatchExportSpanProcessor.worker` thread dominates CPU usage at 100%. See flamegraph:
![image](https://user-images.githubusercontent.com/1356899/95276931-5fca1200-0801-11eb-9136-dd1a422dfa46.png)

`py-spy top` output:
```
Total Samples 3000
GIL: 0.00%, Active: 59.00%, Threads: 2

  %Own   %Total  OwnTime  TotalTime  Function (filename)
 27.00%  59.00%    9.90s    18.99s   worker (trace/export/__init__.py)
  8.00%   8.00%    2.68s     2.68s   _is_owned (threading.py)
 10.00%  10.00%    1.94s     1.94s   _acquire_restore (threading.py)
  7.00%   7.00%    1.84s     1.84s   __enter__ (threading.py)
  5.00%   5.00%    1.45s     1.45s   _release_save (threading.py)
  2.00%   2.00%    1.18s     1.18s   __exit__ (threading.py)
  0.00%   0.00%   0.120s    0.120s   wait (gunicorn/workers/sync.py)
  0.00%  59.00%   0.000s    18.99s   _bootstrap (threading.py)
  0.00%   0.00%   0.000s    0.120s   init_process (gunicorn/workers/base.py)
  0.00%  23.00%   0.000s     6.07s   wait (threading.py)
  0.00%   0.00%   0.000s    0.120s   run_for_one (gunicorn/workers/sync.py)
  0.00%   0.00%   0.000s    0.120s   run (gunicorn/app/base.py)
  0.00%   0.00%   0.000s    0.120s   <module> (gunicorn)
  0.00%   0.00%   0.000s    0.120s   run (gunicorn/workers/sync.py)
  0.00%  59.00%   0.000s    18.99s   _bootstrap_inner (threading.py)
  0.00%   0.00%   0.000s    0.120s   run (gunicorn/app/wsgiapp.py)
  0.00%   0.00%   0.000s    0.120s   spawn_worker (gunicorn/arbiter.py)
  0.00%   0.00%   0.000s    0.120s   manage_workers (gunicorn/arbiter.py)
  0.00%  59.00%   0.000s    18.99s   run (threading.py)
  0.00%  59.00%   0.000s    18.99s   run (sentry_sdk/integrations/threading.py)
  0.00%   0.00%   0.000s    0.120s   run (gunicorn/arbiter.py)
  0.00%   0.00%   0.000s    0.120s   spawn_workers (gunicorn/arbiter.py)
```

**Additional context**
I believe the problem is caused by the [continue statement here](https://github.com/open-telemetry/opentelemetry-python/blob/6943de7d09d4f91ac55d72fb1da4512ab03f3f21/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py#L176), which will skip recalculating the `duration` and `timeout` variables, resulting in the next iteration of the loop using the same `timeout` for `condition.wait` as the previous iteration. In my scenario, taking some thread dumps, I observe that `duration` is set to ~5s, timeout to ~0s, and that these values don't change upon subsequent thread dumps:

```
Thread 100 (active): "Thread-1"
    __enter__ (threading.py:241)
        Arguments::
            self: <Condition at 0x7f56f5e15190>
    worker (trace/export/__init__.py:172)
        Arguments::
            self: <BatchExportSpanProcessor at 0x7f56f5e0ddd0>
        Locals::
            timeout: -0.08887800599999984
            start: +bigint
            end: +bigint
            duration: 5.088878006
    run (threading.py:870)
        Arguments::
            self: <Thread at 0x7f56f5e15390>
    run (sentry_sdk/integrations/threading.py:67)
        Locals::
            a: ()
            kw: {}
            hub: <Hub at 0x7f56f5e15510>
            self: <Thread at 0x7f56f5e15390>
    _bootstrap_inner (threading.py:926)
        Arguments::
            self: <Thread at 0x7f56f5e15390>
    _bootstrap (threading.py:890)
        Arguments::
            self: <Thread at 0x7f56f5e15390>
```


Discussion:
Issue is [also present on master](https://github.com/open-telemetry/opentelemetry-python/blob/master/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py#L216). Should be easily fixed by just resetting the timeout before the `continue` statement.

Just wanted to confirm my logic. I can submit a fix when I get time later!

Resolving patch:
diff --git a/exporter/opentelemetry-exporter-datadog/src/opentelemetry/exporter/datadog/spanprocessor.py b/exporter/opentelemetry-exporter-datadog/src/opentelemetry/exporter/datadog/spanprocessor.py
--- a/exporter/opentelemetry-exporter-datadog/src/opentelemetry/exporter/datadog/spanprocessor.py
+++ b/exporter/opentelemetry-exporter-datadog/src/opentelemetry/exporter/datadog/spanprocessor.py
@@ -119,7 +119,8 @@ def worker(self):
                 with self.condition:
                     self.condition.wait(timeout)
                     if not self.check_traces_queue:
-                        # spurious notification, let's wait again
+                        # spurious notification, let's wait again, reset timeout
+                        timeout = self.schedule_delay_millis / 1e3
                         continue
                     if self.done:
                         # missing spans will be sent when calling flush
diff --git a/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py b/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py
--- a/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py
+++ b/opentelemetry-sdk/src/opentelemetry/sdk/trace/export/__init__.py
@@ -213,7 +213,8 @@ def worker(self):
                     self.condition.wait(timeout)
                     flush_request = self._get_and_unset_flush_request()
                     if not self.queue:
-                        # spurious notification, let's wait again
+                        # spurious notification, let's wait again, reset timeout
+                        timeout = self.schedule_delay_millis / 1e3
                         self._notify_flush_request_finished(flush_request)
                         flush_request = None
                         continue


Test patch:
diff --git a/exporter/opentelemetry-exporter-datadog/tests/test_datadog_exporter.py b/exporter/opentelemetry-exporter-datadog/tests/test_datadog_exporter.py
--- a/exporter/opentelemetry-exporter-datadog/tests/test_datadog_exporter.py
+++ b/exporter/opentelemetry-exporter-datadog/tests/test_datadog_exporter.py
@@ -483,6 +483,42 @@ def test_span_processor_scheduled_delay(self):
 
         tracer_provider.shutdown()
 
+    def test_batch_span_processor_reset_timeout(self):
+        """Test that the scheduled timeout is reset on cycles without spans"""
+        delay = 50
+        # pylint: disable=protected-access
+        exporter = MockDatadogSpanExporter()
+        exporter._agent_writer.write.side_effect = lambda spans: time.sleep(
+            0.05
+        )
+        span_processor = datadog.DatadogExportSpanProcessor(
+            exporter, schedule_delay_millis=delay
+        )
+        tracer_provider = trace.TracerProvider()
+        tracer_provider.add_span_processor(span_processor)
+        tracer = tracer_provider.get_tracer(__name__)
+        with mock.patch.object(span_processor.condition, "wait") as mock_wait:
+            with tracer.start_span("foo"):
+                pass
+
+            # give some time for exporter to loop
+            # since wait is mocked it should return immediately
+            time.sleep(0.1)
+            mock_wait_calls = list(mock_wait.mock_calls)
+
+            # find the index of the call that processed the singular span
+            for idx, wait_call in enumerate(mock_wait_calls):
+                _, args, __ = wait_call
+                if args[0] <= 0:
+                    after_calls = mock_wait_calls[idx + 1 :]
+                    break
+
+            self.assertTrue(
+                all(args[0] >= 0.05 for _, args, __ in after_calls)
+            )
+
+        span_processor.shutdown()
+
     def test_span_processor_accepts_parent_context(self):
         span_processor = mock.Mock(
             wraps=datadog.DatadogExportSpanProcessor(self.exporter)
diff --git a/opentelemetry-sdk/tests/trace/export/test_export.py b/opentelemetry-sdk/tests/trace/export/test_export.py
--- a/opentelemetry-sdk/tests/trace/export/test_export.py
+++ b/opentelemetry-sdk/tests/trace/export/test_export.py
@@ -381,6 +381,43 @@ def test_batch_span_processor_scheduled_delay(self):
 
         span_processor.shutdown()
 
+    def test_batch_span_processor_reset_timeout(self):
+        """Test that the scheduled timeout is reset on cycles without spans"""
+        spans_names_list = []
+
+        export_event = threading.Event()
+        my_exporter = MySpanExporter(
+            destination=spans_names_list,
+            export_event=export_event,
+            export_timeout_millis=50,
+        )
+
+        span_processor = export.BatchExportSpanProcessor(
+            my_exporter, schedule_delay_millis=50,
+        )
+
+        with mock.patch.object(span_processor.condition, "wait") as mock_wait:
+            _create_start_and_end_span("foo", span_processor)
+            self.assertTrue(export_event.wait(2))
+
+            # give some time for exporter to loop
+            # since wait is mocked it should return immediately
+            time.sleep(0.05)
+            mock_wait_calls = list(mock_wait.mock_calls)
+
+            # find the index of the call that processed the singular span
+            for idx, wait_call in enumerate(mock_wait_calls):
+                _, args, __ = wait_call
+                if args[0] <= 0:
+                    after_calls = mock_wait_calls[idx + 1 :]
+                    break
+
+            self.assertTrue(
+                all(args[0] >= 0.05 for _, args, __ in after_calls)
+            )
+
+        span_processor.shutdown()
+
     def test_batch_span_processor_parameters(self):
         # zero max_queue_size
         self.assertRaises(