praekelt/vumi — pull request #646 by smn (2013-11-10 13:12:34+00:00)
Title: SMPP crashes on inbound multipart (CSM) ucs2 messages

```
2013-11-08 16:18:41+0100 [EsmeTransceiver,client] INCOMING <<<< 000000be00000005000000000000003f0001013231383931383236313432320000013130303230004000000000000008008c050003010302064406270646062a062e06270628
002000390039003900390039002e0020062506300627002006430627064600200627064406450631064306320020063a064a063100200635062d064a062d0020062306480020062a0631064a062f0020062306460020062a063a064a063100a0064506310643
06320643060c0020062706440631062c0627
2013-11-08 16:18:41+0100 [EsmeTransceiver,client] INCOMING <<<< {'body': {'mandatory_parameters': {'priority_flag': 0, 'source_addr': '218XXXXXXXXX', 'protocol_id': 0, 'replace_if_present_flag': 0, 'regis
tered_delivery': 0, 'dest_addr_ton': 'unknown', 'source_addr_npi': 'ISDN', 'schedule_delivery_time': '', 'dest_addr_npi': 'ISDN', 'sm_length': 140, 'esm_class': 64, 'data_coding': 8, 'service_type': '', '
source_addr_ton': 'international', 'sm_default_msg_id': 0, 'validity_period': '', 'destination_addr': '10020', 'short_message': "\x05\x00\x03\x01\x03\x02\x06D\x06'\x06F\x06*\x06.\x06'\x06(\x00 \x009\x009\
x009\x009\x009\x00.\x00 \x06%\x060\x06'\x00 \x06C\x06'\x06F\x00 \x06'\x06D\x06E\x061\x06C\x062\x00 \x06:\x06J\x061\x00 \x065\x06-\x06J\x06-\x00 \x06#\x06H\x00 \x06*\x061\x06J\x06/\x00 \x06#\x06F\x00 \x06*
\x06:\x06J\x061\x00\xa0\x06E\x061\x06C\x062\x06C\x06\x0c\x00 \x06'\x06D\x061\x06,\x06'"}}, 'header': {'command_status': 'ESME_ROK', 'command_length': 190, 'sequence_number': 63, 'command_id': 'deliver_sm'
}}
2013-11-08 16:18:41+0100 [EsmeTransceiver,client] OUTGOING >>>> {'body': {'mandatory_parameters': {'message_id': ''}}, 'header': {'command_status': 'ESME_ROK', 'command_length': 17, 'sequence_number': 63,
'command_id': 'deliver_sm_resp'}}
2013-11-08 16:18:41+0100 [EsmeTransceiver,client] Redis multipart key: multi_218XXXXXXXXX_10020_1_3
2013-11-08 16:18:41+0100 [VumiRedis,client] Retrieved value: {u'1': {u'reference_number': 1, u'part_message': u"\x06D\x06J\x06D\x06I\x00 \x069\x06'\x064\x06H\x061\x00 \x06E\x06-\x06E\x06/\x00 \x06C\x061\x
06J\x06(\x06)\x06\x0c\x00 \x06*\x06E\x00 \x06*\x063\x06,\x06J\x06D\x06C\x00 \x06A\x06I\x00 \x06E\x061\x06C\x062\x00 \x001\x00 \x06E\x061\x06C\x062\x00 \x06'\x06D\x06*\x06/\x061\x06J\x06(\x00.\x00 \x061\x0
6B\x06E\x00 \x06E\x061\x06C\x062\x00 \x06'", u'total_number': 3, u'from_msisdn': u'218XXXXXXXXX', u'multipart_type': u'CSM', u'part_number': 1, u'to_msisdn': u'10020'}}
2013-11-08 16:18:41+0100 [VumiRedis,client] Starting factory
2013-11-08 16:18:41+0100 [VumiRedis,client] Unhandled error in Deferred:
2013-11-08 16:18:41+0100 [VumiRedis,client] Starting factory
2013-11-08 16:18:41+0100 [VumiRedis,client] Unhandled Error
Traceback (most recent call last):
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 421, in errback
self._startRunCallbacks(fail)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 488, in _startRunCallbacks
self._runCallbacks()
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 575, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1126, in gotResult
_inlineCallbacks(r, g, deferred)
--- ---
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1068, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/var/praekelt/vumi-libya/ve/src/vumi/vumi/transports/smpp/clientserver/client.py", line 160, in _process_pdu_queue
yield self.handle_data(data)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1068, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/var/praekelt/vumi-libya/ve/src/vumi/vumi/transports/smpp/clientserver/client.py", line 154, in handle_data
yield handler(pdu)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1068, in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "/var/praekelt/vumi-libya/ve/src/vumi/vumi/transports/smpp/clientserver/client.py", line 330, in handle_deliver_sm
yield self._handle_deliver_sm_multipart(pdu, pdu_params)
File "/var/praekelt/vumi-libya/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCallbacks
result = g.send(result)
File "/var/praekelt/vumi-libya/ve/src/vumi/vumi/transports/smpp/clientserver/client.py", line 427, in _handle_deliver_sm_multipart
redis_key, json.dumps(multi.get_array()))
File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
exceptions.UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 109: invalid start byte

2013-11-08 16:18:41+0100 [HTTPPageGetter,client] Stopping factory
2013-11-08 16:18:41+0100 [HTTPPageGetter,client] Stopping factory
(END)
```