Repository: ros/ros_comm

Issue:
Unicode Caller ID prevent receiving topic messages
I am currently investigating this, while diving into rospy code, so I have only a part of the information...

It seems that, whenever `rospy.names.get_name()` returns a `caller_id` that is a unicode string, it then goes into `rospy.impl.tcpros_pubsub.get_header_fields()` which forwards this to `tcpros_base.write_header()` and to `rosgraph.network.write_ros_handshake_header()`.
There `encode_ros_handshake_header` throw `'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)` which is caught in `tcpros_base._tcp_server_callback` and the following message is displayed:

```
[WARN] [WallTime: 1448097800.943432] Inbound TCP/IP connection failed: 'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)
```

And since the connection doesn't happen some communication breaks somewhere.
It took me a very long time to track this down because : 
- There is no information in the error message hinting at where things have gone wrong
- This doesn't break in the main node process, but in the topic subscriber thread, which means that to be able to catch it with a debugger, one needs to actually debug inside rospy itself.

Now here are my two questions :
- what can I do right now to make sure none of my node happen to set the caller_id with a unicode string ? I am currently clueless as to why that caller_id is unicode...
- what can we do to fix this / prevent the error in the long term ?

This happens for me now with https://github.com/asmodehn/rostful but the setup is a bit complicated, identifying the core issue with a simpler setup would be helpful.
I ll keep looking for more clues in the next few days...

Unicode Caller ID prevent receiving topic messages
I am currently investigating this, while diving into rospy code, so I have only a part of the information...

It seems that, whenever `rospy.names.get_name()` returns a `caller_id` that is a unicode string, it then goes into `rospy.impl.tcpros_pubsub.get_header_fields()` which forwards this to `tcpros_base.write_header()` and to `rosgraph.network.write_ros_handshake_header()`.
There `encode_ros_handshake_header` throw `'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)` which is caught in `tcpros_base._tcp_server_callback` and the following message is displayed:

```
[WARN] [WallTime: 1448097800.943432] Inbound TCP/IP connection failed: 'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)
```

And since the connection doesn't happen some communication breaks somewhere.
It took me a very long time to track this down because : 
- There is no information in the error message hinting at where things have gone wrong
- This doesn't break in the main node process, but in the topic subscriber thread, which means that to be able to catch it with a debugger, one needs to actually debug inside rospy itself.

Now here are my two questions :
- what can I do right now to make sure none of my node happen to set the caller_id with a unicode string ? I am currently clueless as to why that caller_id is unicode...
- what can we do to fix this / prevent the error in the long term ?

This happens for me now with https://github.com/asmodehn/rostful but the setup is a bit complicated, identifying the core issue with a simpler setup would be helpful.
I ll keep looking for more clues in the next few days...



Discussion:
I managed to trace down the issue to this place : 

``` python
# check for name override
    mappings = rospy.names.get_mappings()
    print mappings
    if '__name' in mappings:
        name = mappings['__name']
        if anonymous:
            logdebug("[%s] WARNING: due to __name setting, anonymous setting is being changed to false"%name)
            anonymous = False
```

rospy.names.get_mappings() is actually : `{u'__name': u'rostful', u'__log': u'/home/alexv/.ros/log/39179d0e-9181-11e5-aa39-0862664558c8/rostful-3.log'}`
So after this section, name changed from raw string `r'rostful'` (from function param) to unicode `u'rostful'` (from mappings).

This mapping is unicode because the argument is passed from command line directly ( started with roslaunch ).

So It looks like : 
1) the name from command line argument overwrite the name passed as argument in the method.
2) the sys.argv list is not converted/asserted to rawstrings to make sure everything underneath works.
3) rospy.tcpros_pubsub implementation doesn't support unicode strings.

Looking at http://docs.ros.org/api/rospy/html/rospy-module.html#init_node It seems argv should be [str] so that means I need to make that happen on the caller code, before rospy...

In case it s of any help to someone else coming around here, this did the trick : 

``` python
node_str_argv = [unicodedata.normalize('NFKD', arg).encode('ascii', 'ignore') if isinstance(arg, unicode) else str(arg) for arg in node_argv]
rospy.init_node(name=node_name, argv=node_str_argv)
```

I think there is still a piece missing here. You mentioned that the variable was a raw string before and afterwards is a unicode string. While you mentioned that the value was both time just `rostful` the original error message indicates otherwise:

```
'ascii' codec can't decode byte 0xc5
```

Unicode or not Python does convert the string automatically. The problem is that the character is not a character with ascii value 0-127. So the question is what actual value was the problem and where did it come from? Maybe you can provide a reproducible example to answer both questions.

The string is not converted.
The argument ( unicode ) replaces the function param ( str ) as the node name.
Then rospy.tcpros_pubsub assume the name is str ( ascii ), but it is not.

Doing something like `rospy.init_node(r'node_name', argv={u'__name': u'node_name'})` should be enough to trigger the problem...
I ll come up with a detailed example if needed, when i get some time for it.

Using the single line works fine for me. Please provide a [SSCCE](http://sscce.org/).

Sorry my argv syntax was wrong.
To be detected as mapping by init_node, it should have been : `rospy.init_node(r'node_name', argv={u'__name:=node_name'})`

Full Talker node triggering the problem ( plus some boilerplate code to get ROS to initialize from inside python debugger ) : https://gist.github.com/asmodehn/92ff6f9f0cf35a65533c
The warning is displayed and if you plug a listener to it, you will see nothing gets through.
As soon as you remove that unicode arg, things will work.

I managed to trace down the issue to this place : 

``` python
# check for name override
    mappings = rospy.names.get_mappings()
    print mappings
    if '__name' in mappings:
        name = mappings['__name']
        if anonymous:
            logdebug("[%s] WARNING: due to __name setting, anonymous setting is being changed to false"%name)
            anonymous = False
```

rospy.names.get_mappings() is actually : `{u'__name': u'rostful', u'__log': u'/home/alexv/.ros/log/39179d0e-9181-11e5-aa39-0862664558c8/rostful-3.log'}`
So after this section, name changed from raw string `r'rostful'` (from function param) to unicode `u'rostful'` (from mappings).

This mapping is unicode because the argument is passed from command line directly ( started with roslaunch ).

So It looks like : 
1) the name from command line argument overwrite the name passed as argument in the method.
2) the sys.argv list is not converted/asserted to rawstrings to make sure everything underneath works.
3) rospy.tcpros_pubsub implementation doesn't support unicode strings.

Looking at http://docs.ros.org/api/rospy/html/rospy-module.html#init_node It seems argv should be [str] so that means I need to make that happen on the caller code, before rospy...

In case it s of any help to someone else coming around here, this did the trick : 

``` python
node_str_argv = [unicodedata.normalize('NFKD', arg).encode('ascii', 'ignore') if isinstance(arg, unicode) else str(arg) for arg in node_argv]
rospy.init_node(name=node_name, argv=node_str_argv)
```

I think there is still a piece missing here. You mentioned that the variable was a raw string before and afterwards is a unicode string. While you mentioned that the value was both time just `rostful` the original error message indicates otherwise:

```
'ascii' codec can't decode byte 0xc5
```

Unicode or not Python does convert the string automatically. The problem is that the character is not a character with ascii value 0-127. So the question is what actual value was the problem and where did it come from? Maybe you can provide a reproducible example to answer both questions.

The string is not converted.
The argument ( unicode ) replaces the function param ( str ) as the node name.
Then rospy.tcpros_pubsub assume the name is str ( ascii ), but it is not.

Doing something like `rospy.init_node(r'node_name', argv={u'__name': u'node_name'})` should be enough to trigger the problem...
I ll come up with a detailed example if needed, when i get some time for it.

Using the single line works fine for me. Please provide a [SSCCE](http://sscce.org/).

Sorry my argv syntax was wrong.
To be detected as mapping by init_node, it should have been : `rospy.init_node(r'node_name', argv={u'__name:=node_name'})`

Full Talker node triggering the problem ( plus some boilerplate code to get ROS to initialize from inside python debugger ) : https://gist.github.com/asmodehn/92ff6f9f0cf35a65533c
The warning is displayed and if you plug a listener to it, you will see nothing gets through.
As soon as you remove that unicode arg, things will work.


Resolving patch:
diff --git a/tools/rosgraph/src/rosgraph/network.py b/tools/rosgraph/src/rosgraph/network.py
--- a/tools/rosgraph/src/rosgraph/network.py
+++ b/tools/rosgraph/src/rosgraph/network.py
@@ -400,11 +400,13 @@ def encode_ros_handshake_header(header):
     # in the usual configuration, the error 'TypeError: can't concat bytes to str' appears:
     if python3 == 0:
         #python 2
+        fields = [str(f) for f in fields]
         s = ''.join(["%s%s"%(struct.pack('<I', len(f)), f) for f in fields])
         return struct.pack('<I', len(s)) + s
     else:
         #python 3 
-        s = b''.join([(struct.pack('<I', len(f)) + f.encode("utf-8")) for f in fields])
+        fields = [f.encode('utf-8') for f in fields]
+        s = b''.join([(struct.pack('<I', f) + f) for f in fields])
         return struct.pack('<I', len(s)) + s
                                         
 def write_ros_handshake_header(sock, header):


Test patch: