Repository: dmwm/wmcore
Issue: [py2py3] Possible bug in ConfigSectionTree.format()

## How to reproduce it

We have not seen any bug yet, but it is possible that the changes to `ConfigSectionTree` introduced in #10103 have some side effect.

## Describe the bug

Before #10103 we had

```python
def formatNative(value):
    if isinstance(value, int):
        return value
    if isinstance(value, float):
        return value
    if isinstance(value, list):
        return value
    if isinstance(value, dict):
        return value # this was actually "return dict", but it is outside of our scope now
    else:
        return format(value)

def format(value):
    if isinstance(value, str):
        value = "\\'%s\\'" % value
    return str(value)
```

Since we run this code in py2 only, only py2-`str` would get surrounded in `\\'`, not unicode strings, and the return type of this function would be py2-`str` (i.e. bytes string)

In #10103 we introduced `from builtins import str`. This means that right now, when we run the code in python2, py2-`str` are not surrounded by `\\'`, instead unicode strings are, and the return type of the function is unicode, not bytes.

We have effectively changed the API of `ConfigSectionTree` without noticing, which can have negative repercussions.

## Impact of the bug

Is this change a real problem? I am not sure. `formatNative` seems to be used only in `WMCore/WMSpec/WMTask.WMTaskHelper.getGeneratorSettings` in

https://github.com/dmwm/WMCore/blob/08c1eea1a3245ebf956438e0a1ab1b79e338ff40/src/python/WMCore/WMSpec/WMTask.py#L631

Which is then used in `WMCore/JobSplitting/Generators/GeneratorManager.GeneratorManager` and `WMCore/JobSplitting/Generators/GeneratorFacotory.GeneratorFactory.makeGenerators`.

## Expected behavior

Before #10103 we had

```python
def format(value):
    if isinstance(value, str):
        value = "\\'%s\\'" % value
    return str(value)
```

This means that when this code is run in py2 only py2-`str` (sequence of bytes) are surrounded by quotes and that this function returns py2-`str` (sequence of bytes).

Do we want to just use `str` in both py2 and py3 even though they mean different things? Or do we absolutely need to be consistent in both py2 and py3 and always return the same string type in both py2 and py3? It is not clear to me which is the best option at this moment, this is the kind of problem whose investigation would greatly benefit from a py3 environment.

If someone has a better understanding than me, then feel free to comment! Any help is much appreciated!

## Possible approaches

We have a couple of options and I am not sure which is the best, since it is a bit hard for me to grasp how the "pythonisedDict" is used in the Generators.

#### 1. Leave it as is (surround unicode and return unicode)

and investigate in testbed the impact of this problem.

(This may not be the best option)

#### 2. Surround both unicode and bytes, return unicode

We could change 

```python
- from builtins import str, object
+ from builtins import str, bytes, object

def format(value):
-    if isinstance(value, str):
+    if isinstance(value, (str, bytes)):
        value = "\\'%s\\'" % value
+    return str(value)
```

(This is a nice option)

#### 3. Make `ConfigSectionTree` behave the same as before #10103 (surround bytes and return bytes)

We could change 

```python
- from builtins import str, object
+ from builtins import str, bytes, object

def format(value):
-    if isinstance(value, str):
+    if isinstance(value, bytes):
        value = "\\'%s\\'" % value
-    return str(value)
+    return bytes(value)
```

This may not be what we want, though, especially in python3.

#### 4. Try the "native string" approach

This is inspired by the Fedora's Conservative Py3 Porting Guide https://portingguide.readthedocs.io/en/latest/strings.html#the-native-string

The idea is: often, when we use `str` in py2, we are just consistently using `str` and we may be better off to just forget about unicode,bytes and other amenities and just keep using `str` in both py2 and py3, even though it means different things.

when run from py2 interpreter:
- surround bytes, return bytes
when run from py3 interpreter:
- surround unicode, return unicode

(This is a nice option)


Changed files:
src/python/WMCore/WMSpec/ConfigSectionTree.py

diff --git a/src/python/WMCore/WMSpec/ConfigSectionTree.py b/src/python/WMCore/WMSpec/ConfigSectionTree.py
index 3160565f0..0c7751845 100644
--- a/src/python/WMCore/WMSpec/ConfigSectionTree.py
+++ b/src/python/WMCore/WMSpec/ConfigSectionTree.py
@@ -10,7 +10,7 @@ of ConfigSections
 
 
 
-from builtins import str, object
+from builtins import str, bytes, object
 
 from WMCore.Configuration import ConfigSection
 
@@ -227,7 +227,7 @@ def format(value):
     format a value as python
     keep parameters simple, trust python...
     """
-    if isinstance(value, str):
+    if isinstance(value, (str, bytes)):
         value = "\\'%s\\'" % value
     return str(value)