Repository: StackStorm/st2

Issue:
ST2 logs writing "(unknown file)" in log prefix, instead of module name
## SUMMARY

In ST2 v3.2, the st2sensorcontainer.log, and in v3.3 all st2 logs have the `module` name replaced with `(unknown file)`


### STACKSTORM VERSION

v3.2, v3.3, EL8 and U18.04
Python 3.6

### OS, environment, install method

One Line and Ansible so far. 

## Steps to reproduce the problem

Show how to reproduce the problem, using a minimal test-case. Make sure to include any content
(pack content - workflows, actions, etc.) which are needed to reproduce the problem.

## Expected Results

No `(unknown file)`, but the python module name. 

## Actual Results
```
2020-10-13 17:55:13,787 140460262501416 INFO (unknown file) [-] Sensor linux.FileWatchSensor started
2020-10-13 17:55:15,444 139725927337456 INFO (unknown file) [-] No config found for sensor "FileWatchSensor"
2020-10-13 17:55:15,446 139725927337456 INFO (unknown file) [-] Watcher started
2020-10-13 17:55:15,446 139725927337456 INFO (unknown file) [-] Running sensor initialization code
2020-10-13 17:55:15,454 139725927337456 INFO (unknown file) [-] Running sensor in passive mode
```




Discussion:
This is python 3 only. The logging module determines this information using a `logging.findCaller()` function.

Python 2.7 version: https://docs.python.org/2/library/logging.html#logging.Logger.findCaller
Python 3.6 version: https://docs.python.org/3.6/library/logging.html#logging.Logger.findCaller

StackStorm overrides this function here: https://github.com/StackStorm/st2/blob/master/st2common/st2common/log.py#L131
Our implementation is here: https://github.com/StackStorm/st2/blob/master/st2common/st2common/log.py#L72-L95

This function has the signature for the 2.7 python version and returns a 3-element tuple as python 2 expects. However, in Python 3 a new argument is specified called `stack_info` and the returned value from `logging.findCaller()` is expected to be a 4-element tuple. This 4th element is stack information derived from the `stack_info` value passed in. 

We currently do not handle these cases and are returning an incorrectly sized tuple from the function in python 3.

Resolving patch:
diff --git a/st2common/st2common/log.py b/st2common/st2common/log.py
--- a/st2common/st2common/log.py
+++ b/st2common/st2common/log.py
@@ -15,6 +15,7 @@
 
 from __future__ import absolute_import
 
+import io
 import os
 import sys
 import logging
@@ -69,25 +70,64 @@
 _srcfile = get_normalized_file_path(__file__)
 
 
-def find_caller(*args, **kwargs):
+def find_caller(stack_info=False, stacklevel=1):
     """
     Find the stack frame of the caller so that we can note the source file name, line number and
     function name.
 
     Note: This is based on logging/__init__.py:findCaller and modified so it takes into account
-    this file - https://hg.python.org/cpython/file/2.7/Lib/logging/__init__.py#l1233
+    this file:
+    https://github.com/python/cpython/blob/2.7/Lib/logging/__init__.py#L1240-L1259
+
+    The Python 3.x implementation adds in a new argument `stack_info` and `stacklevel`
+    and expects a 4-element tuple to be returned, rather than a 3-element tuple in
+    the python 2 implementation.
+    We derived our implementation from the Python 3.9 source code here:
+    https://github.com/python/cpython/blob/3.9/Lib/logging/__init__.py#L1502-L1536
+
+    We've made the appropriate changes so that we're python 2 and python 3 compatible depending
+    on what runtine we're working in.
     """
-    rv = '(unknown file)', 0, '(unknown function)'
+    if six.PY2:
+        rv = '(unknown file)', 0, '(unknown function)'
+    else:
+        # python 3, has extra tuple element at the end for stack information
+        rv = '(unknown file)', 0, '(unknown function)', None
 
     try:
-        f = logging.currentframe().f_back
+        f = logging.currentframe()
+        # On some versions of IronPython, currentframe() returns None if
+        # IronPython isn't run with -X:Frames.
+        if f is not None:
+            f = f.f_back
+        orig_f = f
+        while f and stacklevel > 1:
+            f = f.f_back
+            stacklevel -= 1
+        if not f:
+            f = orig_f
+
         while hasattr(f, 'f_code'):
             co = f.f_code
             filename = os.path.normcase(co.co_filename)
             if filename in (_srcfile, logging._srcfile):  # This line is modified.
                 f = f.f_back
                 continue
-            rv = (filename, f.f_lineno, co.co_name)
+
+            if six.PY2:
+                rv = (filename, f.f_lineno, co.co_name)
+            else:
+                # python 3, new stack_info processing and extra tuple return value
+                sinfo = None
+                if stack_info:
+                    sio = io.StringIO()
+                    sio.write('Stack (most recent call last):\n')
+                    traceback.print_stack(f, file=sio)
+                    sinfo = sio.getvalue()
+                    if sinfo[-1] == '\n':
+                        sinfo = sinfo[:-1]
+                    sio.close()
+                rv = (filename, f.f_lineno, co.co_name, sinfo)
             break
     except Exception:
         pass


Test patch: