Repository: open-telemetry/opentelemetry-python

Issue:
Default otel reporter timeout of 900s is too long.
Since this timeout can occur when a process is closing/exiting it can mean a significant amount of time when the process is just retrying before it ultimately ends.

For example, it's easy to misconfigure the target service e.g.:

```
	status = StatusCode.UNAVAILABLE
	details = "DNS resolution failed for service: opencensus-collector:55680"
```

which can cause processes to easily hang at exit (and print nothing by default).


Discussion:
I agree it is probably a bit too much but reducing it should not be the solution to services freezing at shutdown. At shutdown, exporters/processors should be given a grace-period to shutdown cleanly and if they don't then the pipeline should force them to shutdown and exit. We should have a different issue for that. 
@owais 
What is the decision for this issue? Not increase the timeout but have a better shutdown mechanism for exporters/processors?
I think these are two separate issues. 

1. Exporter should not hang om shutdown and should have a grace period after which it force quits dropping in memory spans. 
2. Exporter should have a shorter timeout on network ops in general as 900s is probably too long unless this is not a real problem and only manifests when network errors happen during shutdown. In that case we only need 1.

I think we can address both. Java has default [timeout](https://github.com/open-telemetry/opentelemetry-java/blob/43f1ecf7ede49104c2aac1c9e5448b7cf7859ce7/exporters/otlp/trace/src/main/java/io/opentelemetry/exporter/otlp/trace/OtlpGrpcSpanExporterBuilder.java#L31) to be 10 seconds, maybe we can do the same?
Related https://github.com/open-telemetry/opentelemetry-python/issues/346
SGTM
@lzchen OTLP python also has the default timeout 10 seconds for Export. 
https://github.com/open-telemetry/opentelemetry-python/blob/24edd3d81afb394c3b2d1f0416faca59db931b66/exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py#L183-L187
The 900 secs is to deal with the transient errors by using exponential back-off retry strategy. This might still be too long but just want to make sure we are not confusing one with other. https://github.com/open-telemetry/opentelemetry-python/blob/24edd3d81afb394c3b2d1f0416faca59db931b66/exporter/opentelemetry-exporter-otlp/src/opentelemetry/exporter/otlp/exporter.py#L247-L298

@lonewolf3739 
Thanks for the explanation. It's outlined that it is the same value as the Go implementation. If that's the case shouldn't we be consistent? Does 900s make sense for the "conventional" use case of actually retrying instead of this specific use case where the exporter hangs due to misconfiguration? If so, we should just solve (1) as outlined [here](https://github.com/open-telemetry/opentelemetry-python/issues/1670#issuecomment-792964513).
900 secs is too high for maximum backoff time. Generally it is either 32 or 64 seconds and sometimes can be little high based on use case. I couldn't find what is the recommended maximum value by OTLP but I assume it wouldn't be as high 900 seconds. Probably we can raise an issue on the spec repo.
Thanks for digging into this @lonewolf3739. Agree we should clarify with spec or at least with other SIG maintainers. 

And we should probably have separate tickets for possibly reducing exp-backoff max time and add grace period to exporters on shutdown.
This issue was marked stale due to lack of activity. It will be closed in 30 days.
I brought this up in spec sig meeting sometime back and they agreed this is too long but wouldn't recommend any value now as it requires some work to do. So as of now it is upto language implementations to decide.
@lonewolf3739 
So is the recommendation to leave it as it is?
> @lonewolf3739
> So is the recommendation to leave it as it is?

Until the spec is updated it is upto each language SIG to make decision on what to do.
@lonewolf3739 
So for now can we just reduce it to 64s or something?
Yes, I think that is very reasonable value.
Assigning self to address this and related issues.
Please let me know on when is this planned to be changed, ready to help with this change.

Resolving patch:
diff --git a/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py b/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py
--- a/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py
+++ b/exporter/opentelemetry-exporter-otlp-proto-grpc/src/opentelemetry/exporter/otlp/proto/grpc/exporter.py
@@ -262,14 +262,11 @@ def _translate_data(
         pass
 
     def _export(self, data: TypingSequence[SDKDataT]) -> ExportResultT:
+
+        max_value = 64
         # expo returns a generator that yields delay values which grow
         # exponentially. Once delay is greater than max_value, the yielded
         # value will remain constant.
-        # max_value is set to 900 (900 seconds is 15 minutes) to use the same
-        # value as used in the Go implementation.
-
-        max_value = 900
-
         for delay in expo(max_value=max_value):
 
             if delay == max_value:
@@ -289,8 +286,6 @@ def _export(self, data: TypingSequence[SDKDataT]) -> ExportResultT:
                 if error.code() in [
                     StatusCode.CANCELLED,
                     StatusCode.DEADLINE_EXCEEDED,
-                    StatusCode.PERMISSION_DENIED,
-                    StatusCode.UNAUTHENTICATED,
                     StatusCode.RESOURCE_EXHAUSTED,
                     StatusCode.ABORTED,
                     StatusCode.OUT_OF_RANGE,


Test patch: