Repository: rj79/pynetstring

Issue:
Leading zeroes before empty netstrings
Hello!

Thank you for your project! It appeared to be very useful.

[Specification of netstring format](http://cr.yp.to/proto/netstrings.txt) is not very clear about leading zeroes. It makes statements and provides few examples without specifying whether leading zeroes rules are limited to these examples or leading zero rule is universal:

> Any string of 8-bit bytes may be encoded as [len]":"[string]",".
   Here [string] is the string and [len] is a nonempty sequence of ASCII
   digits giving the length of [string] in decimal. The ASCII digits are
   <30> for 0, <31> for 1, and so on up through <39> for 9. Extra zeros
   at the front of [len] are prohibited: [len] begins with <30> exactly
   when [string] is empty.
> 
>  For example, the string "hello world!" is encoded as <31 32 3a 68
   65 6c 6c 6f 20 77 6f 72 6c 64 21 2c>, i.e., "12:hello world!,". The
   empty string is encoded as "0:,".

It's hard to say from specification whether netstring `b"000000000:,"` is allowed or not. [Wikipedia article on Netstring](https://en.wikipedia.org/wiki/Netstring) states:

> The length is written without leading zeroes. Empty string is the only netstring that begins with zero. There is exactly one legal netstring encoding for any byte string.

This particular restriction about only one legal netstring representation vouches for prohibiting leading zeroes in all cases. However, I could not find any reliable source which backs this statement.

Current code allows leading zeroes before empty netstring. It allows unrestricted bloat of input buffer with zeroes. I see two approaches to handle this:
* Restrict all `[len]` from having leading zero. Pros: faster, follows idea of unique netstring representation for each bytestring. Cons: rejects inputs generated by ill applications.
* Silently consume leading zeroes when parsing length, saving only meaningful data into input buffer. This will require additional logic on each length parsing step, but it'll work in all cases without potential input buffer bloat. Pros: follows classic [Robustness principle](https://en.wikipedia.org/wiki/Robustness_principle) "be conservative in what you do, be liberal in what you accept from others.". Cons: slower.

What do you think, which approach is correct one?

If you need help with implementing either approach, please indicate and I'll try to make PR with solution.

Comments:

Patch:
diff --git a/README.rst b/README.rst
index 2a1c20c..010528f 100644
--- a/README.rst
+++ b/README.rst
@@ -57,6 +57,16 @@ The receiving side could look something like this:
     for item in decoded_list:
         print(item)
 
+Also Decoder class supports limit on maximal decoded netstring length. It is
+required for network applications to restrict maximal length of input buffer
+in order to prevent memory bloat. Netstring length limit specified as first 
+argument in this case:
+::
+  decoder = pynetstring.Decoder(maxlen=1024)
+
+Decoder will raise TooLong exception as soon as it'll discover next string
+can't fit limit.
+
 Data encoding
 -------------
 Regardless of the type of the data that is sent to encode(), it will always
@@ -78,11 +88,16 @@ e.g. UTF-8, you have to explicitly do that conversion.
 
 Error handling
 --------------
-A ValueError exception will be raised if trying to decode an invalid 
+A ParseError subclass exception will be raised if trying to decode an invalid 
 netstring.
 ::
- # ValueError exception due to leading zero in non-empty netstring:
- pynetstring.decode('01:A,')
+ # IncompleteString exception due to missing trailing comma:
+ pynetstring.decode('3:ABC_')
+
+ decoder = Decoder(3)
+ # TooLong exception due to exceeded netstring limit in stream parser:
+ decoder.feed(b'4:ABCD,')
+
 
- # ValueError exception due to missing trailing comma:
- pynetstring.decode('3:ABC_')
\ No newline at end of file
+All other exceptions of this module can be expected to be subclass of 
+NetstringException.
diff --git a/pynetstring.py b/pynetstring.py
index a461e15..29dfa72 100644
--- a/pynetstring.py
+++ b/pynetstring.py
@@ -1,4 +1,5 @@
 from enum import Enum
+import math
 
 def _encode(data):
     if isinstance(data, str):
@@ -11,17 +12,35 @@ def encode(data):
     return _encode(data)
 
 
+class NetstringException(Exception):
+    pass
+
+
+class ParseError(NetstringException):
+    pass
+
+
+class TooLong(ParseError):
+    pass
+
+
+class IncompleteString(ParseError):
+    pass
+
+
 class State(Enum):
     PARSE_LENGTH = 0
     PARSE_DATA = 1
 
 
 class Decoder:
-    def __init__(self):
+    def __init__(self, maxlen=0):
         self._buffer = bytearray()
         self._state = State.PARSE_LENGTH
         self._length = None
         self._decoded = []
+        self._maxlen = maxlen
+        self._maxlen_bytes = math.log10(self._maxlen) + 1 if self._maxlen else 0
 
     def feed(self, data):
 
@@ -35,11 +54,17 @@ class Decoder:
                 length_end = self._buffer.find(b':')
                 if length_end == -1:
                     # There is not enough data yet to decode the length
+                    # Compact leading zeroes in buffer
+                    self._buffer = self._buffer[:-1].lstrip(b'0') + self._buffer[-1:]
+                    # Check if length field already too long:
+                    if self._maxlen_bytes and len(self._buffer) > self._maxlen_bytes:
+                        raise TooLong("Read too many bytes but length end"
+                                      " marker wasn't met.")
                     break
                 else:
                     self._length = int(self._buffer[:length_end])
-                    if self._length > 0 and self._buffer[0] == ord(b'0'):
-                        raise ValueError('Leading zero in non-empty netstring.')
+                    if self._maxlen and self._length > self._maxlen:
+                        raise TooLong("Specified netstring length is over limit.")
                     # Consume what has been parsed
                     self._buffer = self._buffer[length_end + 1:]
                     self._state = State.PARSE_DATA
@@ -51,7 +76,7 @@ class Decoder:
                     break
                 else:
                     if self._buffer[self._length] != ord(b','):
-                        raise ValueError('Missing trailing comma.')
+                        raise IncompleteString('Missing trailing comma.')
                     data = self._buffer[:self._length]
                     self._buffer = self._buffer[self._length + 1:]
                     self._decoded.append(bytes(data))


Test patch:
diff --git a/tests/tests.py b/tests/tests.py
index 20343d7..2c5fa77 100644
--- a/tests/tests.py
+++ b/tests/tests.py
@@ -7,15 +7,14 @@ class TestNetString(unittest.TestCase):
         self.assertEqual(b'0:,', netstring.encode(''))
         self.assertEqual(b'0:,', netstring.encode(b''))
 
-    def test_leading_zero_in_length_raises_exception_if_data_nonempty(self):
-        with self.assertRaises(ValueError):
-            netstring.decode('01:X,')
+    def test_accept_leading_zero(self):
+        self.assertEqual([b'X'], netstring.decode('01:X,'))
 
     def test_decode_empty_string(self):
         self.assertEqual([b''], netstring.decode(b'0:,'))
 
     def test_decode_missing_comma_fails(self):
-        with self.assertRaises(ValueError):
+        with self.assertRaises(netstring.IncompleteString):
             netstring.decode('3:abc_')
 
     def test_encode_one_byte_string(self):
@@ -23,7 +22,7 @@ class TestNetString(unittest.TestCase):
         self.assertEqual(b'1:X,', netstring.encode(b'X'))
 
     def test_decode_nested(self):
-        self.assertEqual([b'5:Hello,6:World!'], netstring.decode('16:5:Hello,6:World!,,'))
+        self.assertEqual([b'5:Hello,6:World!,'], netstring.decode('17:5:Hello,6:World!,,'))
 
     def test_feed_empty_string_partially(self):
         decoder = netstring.Decoder()
@@ -43,3 +42,42 @@ class TestNetString(unittest.TestCase):
 
     def test_encode_sequence(self):
         self.assertEqual([b'3:foo,', b'3:bar,'], netstring.encode(['foo', 'bar']))
+
+    def test_no_limit_error_when_content_shorter(self):
+        decoder = netstring.Decoder(10)
+        try:
+            self.assertEqual([b'XXXXXXXXX'], decoder.feed(b'9:XXXXXXXXX,'))
+        except netstring.TooLong:
+            self.assertFalse(True)
+
+    def test_no_limit_error_when_content_just_right(self):
+        decoder = netstring.Decoder(1)
+        try:
+            self.assertEqual([b'X'], decoder.feed(b'1:X,'))
+        except netstring.TooLong:
+            self.assertFalse(True)
+
+        decoder = netstring.Decoder(10)
+        try:
+            self.assertEqual([b'XXXXXXXXXX'], decoder.feed(b'10:XXXXXXXXXX,'))
+        except netstring.TooLong:
+            self.assertFalse(True)
+
+    def test_limit_error_when_content_too_long(self):
+        decoder = netstring.Decoder(1)
+        with self.assertRaises(netstring.TooLong):
+            decoder.feed(b'2:XX,')
+
+    def test_limit_error_when_missing_comma(self):
+        decoder = netstring.Decoder(10)
+        with self.assertRaises(netstring.IncompleteString):
+            decoder.feed(b'10:XXXXXXXXXXX,')
+
+    def test_limit_error_when_length_declaration_inplausibly_long(self):
+        decoder = netstring.Decoder(100)
+        with self.assertRaises(netstring.TooLong):
+            # The length declaration of a netstring with content of 100 bytes
+            # takes up at most log10(100) + 1 = 3 bytes.
+            # Error raised already here because 4 bytes of length parsed but
+            # no length end marker ":" found yet.
+            decoder.feed(b'1000')