Repository: dmwm/wmcore
Issue: [py2py3] Make DBFormatter return native strings instead of bytes

**Impact of the bug**
Mostly WMAgent (and externals)

**Describe the bug**
Some places in WMCore actually expect a python3 string data type, instead of bytes, as it's been reported in this comment:
https://github.com/dmwm/WMCore/issues/10634#issuecomment-869847004

StompAMQ was also failing in AgentStatusWatcher with:
```
2021-06-28 18:20:05,645:140147165435648:ERROR:StompAMQ:Notification: {'type': 'cms_wmagent_info', 'version': '0.3', 'producer': 'wmagent'} (type=<class 'dict'>) not send, error: Object of type bytes is not JSON serializable
```

So, I believe it would be better to actually format the DAOs results to native string instead of encoding its output to bytes.

**How to reproduce it**
Run WMAgent Python3 `1.5.0.pre3`, with some patches on top of it to allow components to be started.

**Expected behavior**
Instead of casting strings/bytes to byte strings in the DBFormatter `formatDict` and `formatList`, I'd suggest to actually decode the byte/string output to unicode (native string in python2 and python3).
It fixes at least the problem reported here: https://github.com/dmwm/WMCore/pull/10635#issuecomment-869897102

I think the same change should actually get extended to the other methods formatting DAOs results. But since we only have the current bytes casting in those two methods, I think it's a good start.

**Additional context and error message**
None


Changed files:
src/python/WMCore/Database/DBFormatter.py

diff --git a/src/python/WMCore/Database/DBFormatter.py b/src/python/WMCore/Database/DBFormatter.py
index be7eb5f39..587571515 100644
--- a/src/python/WMCore/Database/DBFormatter.py
+++ b/src/python/WMCore/Database/DBFormatter.py
@@ -6,14 +6,15 @@ Holds a bunch of helper methods to format input and output of sql
 interactions.
 """
 
-from builtins import str, zip, range
+from builtins import str, bytes, zip, range
 
 import datetime
 import time
 import types
 
+from Utils.Utilities import decodeBytesToUnicodeConditional
 from WMCore.DataStructs.WMObject import WMObject
-from Utils.PythonVersion import PY2
+from Utils.PythonVersion import PY3
 
 
 class DBFormatter(WMObject):
@@ -78,10 +79,14 @@ class DBFormatter(WMObject):
                 entry = {}
                 for index in range(0, len(descriptions)):
                     # WARNING: Oracle returns table names in CAP!
-                    if isinstance(i[index], str) and PY2:
-                        entry[str(descriptions[index].lower())] = i[index].encode("utf-8")
+                    if isinstance(descriptions[index], (str, bytes)):
+                        keyName = decodeBytesToUnicodeConditional(descriptions[index], condition=PY3)
                     else:
-                        entry[str(descriptions[index].lower())] = i[index]
+                        keyName = descriptions[index]
+                    if isinstance(i[index], (str, bytes)):
+                        entry[keyName.lower()] = decodeBytesToUnicodeConditional(i[index], condition=PY3)
+                    else:
+                        entry[keyName.lower()] = i[index]
 
                 dictOut.append(entry)
 
@@ -99,8 +104,8 @@ class DBFormatter(WMObject):
             descriptions = r.keys
             for i in r.fetchall():
                 for index in range(0, len(descriptions)):
-                    if isinstance(i[index], str):
-                        listOut.append(i[index].encode("utf-8"))
+                    if isinstance(i[index], (str, bytes)):
+                        listOut.append(decodeBytesToUnicodeConditional(i[index], condition=PY3))
                     else:
                         listOut.append(i[index])
             r.close()