Repository: DataDog/dd-trace-py

Issue:
TypeError: '<' not supported between instances of 'NoneType' and 'str'
### Infos
We're using DD to monitor our systems. From time to time we see the following errors popping up in our Sentry environment.

Maybe related to #3019?

### Which version of dd-trace-py are you using?
0.56.0

### Which version of pip are you using?
latest

### What is the result that you get?
``` 
TypeError: '<' not supported between instances of 'NoneType' and 'str'
  File "ddtrace/profiling/scheduler.py", line 53, in flush
    exp.export(events, start, self._last_export)
  File "ddtrace/profiling/exporter/http.py", line 150, in export
    profile = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)
  File "ddtrace/profiling/exporter/pprof.pyx", line 630, in ddtrace.profiling.exporter.pprof.PprofExporter.export
  File "ddtrace/profiling/exporter/pprof.pyx", line 314, in ddtrace.profiling.exporter.pprof._PprofConverter._build_profile
```


Discussion:
We just had a similar issue on version `0.56.1`, I've also updaed in #3019 .
Backport for 0.56 here: https://github.com/DataDog/dd-trace-py/pull/3062
We are still having the error in 0.57:
```
File "ddtrace/profiling/exporter/pprof.pyx", line 594, in ddtrace.profiling.exporter.pprof.PprofExporter.export
File "ddtrace/profiling/exporter/pprof.pyx", line 493, in ddtrace.profiling.exporter.pprof.PprofExporter._group_stack_events
TypeError: '<' not supported between instances of 'int' and 'NoneType'
```
> 

Ok, this is weird. Line 493 is:

```
sorted(events, key=self._stack_event_group_key),
```

The error makes since comparison is being used here.
What's weird is that `_stack_event_group_key` returns a `StackEventGroupKey` whose typing has no `None`:

```python
StackEventGroupKey = typing.NamedTuple(
    "StackEventGroupKey",
    [
        ("thread_id", str),
        ("thread_native_id", str),
        ("thread_name", str),
        ("task_id", str),
        ("task_name", str),
        ("local_root_span_id", str),
        ("span_id", str),
        ("trace_resource", str),
        ("trace_type", str),
        ("frames", HashableStackTraceType),
        ("nframes", int),
    ],
)
```

Only `nframes` is an `int`, and worst case it's set to 0 — never `None` as far as I can tell glancing at the code.
`_stack_event_group_key` does convert all data to the right type.

@Apakottur do you see 100% of the time or randomly from time to time?
@jd I see it randomly from time to time, I think.
I don't see any impact on APM as well, just this error popping on Sentry.
We've had 4 occurences of the error in the last few days, and I was using APM to analyze thousands of requests successfully.

I can't see the contents of `self._stack_event_group_key` on Sentry, but I do have some details from
```
ddtrace/profiling/scheduler.py:56
exp.export(events, start, self._last_export)
```
Which calls 
```
ddtrace.profiling.exporter.pprof.PprofExporter.export
```

```
{
<class 'ddtrace.profiling.collector.memalloc.MemoryAllocSampleEvent'>: [
MemoryAllocSampleEvent(timestamp=1640358086011282246, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 156, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile')..., 
MemoryAllocSampleEvent(timestamp=1640358086011320577, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 156, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 168, '_compile')..., 
MemoryAllocSampleEvent(timestamp=1640358086011348129, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 156, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 168, '_compile')..., 
MemoryAllocSampleEvent(timestamp=1640358086011374247, sampling_period=None, thread_id=140070060222272, thread_name='MainThread', thread_native_id=1, task_id=None, task_name=None, frames=(('/usr/local/lib/python3.10/sre_compile.py', 311, '_optimize_charset'), ('/usr/local/lib/python3.10/sre_compile.py', 120, '_compile'), ('/usr/local/lib/python3.10/sre_compile.py', 209, '_compile'), ('/usr/local/lib/python3.10/sre...
]
}
```

Maybe this helps?
That actually gave me an idea. You might have a stack trace where `lineno` is `None`. That should not happen, but who knows, it's possible that in some cases (maybe `eval`?) the lineno is set to `None` and then comparison would fail:

```python
>>> stack1 = [("somefile", 1, "funcname")]
>>> stack2 = [("somefile", None, "funcname")]
>>> stack1 < stack2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'NoneType'
```

I'll write up a PR to fix that and make sure all line numbers that are `None` are actually set to `0`.

Resolving patch:
diff --git a/ddtrace/profiling/collector/threading.py b/ddtrace/profiling/collector/threading.py
--- a/ddtrace/profiling/collector/threading.py
+++ b/ddtrace/profiling/collector/threading.py
@@ -7,6 +7,7 @@
 
 import attr
 
+import ddtrace
 from ddtrace.internal import compat
 from ddtrace.internal import nogevent
 from ddtrace.internal.utils import attr as attr_utils
@@ -19,6 +20,10 @@
 from ddtrace.vendor import wrapt
 
 
+if typing.TYPE_CHECKING:
+    from ddtrace.profiling import recorder as ddrecorder
+
+
 @event.event_class
 class LockEventBase(event.StackBasedEvent):
     """Base Lock event."""
@@ -62,7 +67,16 @@ def _current_thread():
 
 
 class _ProfiledLock(wrapt.ObjectProxy):
-    def __init__(self, wrapped, recorder, tracer, max_nframes, capture_sampler, endpoint_collection_enabled):
+    def __init__(
+        self,
+        wrapped,  # type: threading.Lock
+        recorder,  # type: ddrecorder.Recorder
+        tracer,  # type: ddtrace.Tracer
+        max_nframes,  # type: int
+        capture_sampler,  # type: collector.CaptureSampler
+        endpoint_collection_enabled,  # type: bool
+    ):
+        # type: (...) -> None
         wrapt.ObjectProxy.__init__(self, wrapped)
         self._self_recorder = recorder
         self._self_tracer = tracer
@@ -71,7 +85,10 @@ def __init__(self, wrapped, recorder, tracer, max_nframes, capture_sampler, endp
         self._self_endpoint_collection_enabled = endpoint_collection_enabled
         frame = sys._getframe(2 if WRAPT_C_EXT else 3)
         code = frame.f_code
-        self._self_name = "%s:%d" % (os.path.basename(code.co_filename), frame.f_lineno)
+        self._self_name = "%s:%d" % (
+            os.path.basename(code.co_filename),
+            0 if frame.f_lineno is None else frame.f_lineno,
+        )
 
     def acquire(self, *args, **kwargs):
         if not self._self_capture_sampler.capture():
diff --git a/ddtrace/profiling/event.py b/ddtrace/profiling/event.py
--- a/ddtrace/profiling/event.py
+++ b/ddtrace/profiling/event.py
@@ -9,7 +9,7 @@
 _T = typing.TypeVar("_T")
 
 # (filename, line number, function name)
-FrameType = typing.Tuple[str, int, typing.Optional[str]]
+FrameType = typing.Tuple[str, int, str]
 StackTraceType = typing.List[FrameType]
 
 


Test patch: