Repository: DataBiosphere/toil

Issue:
AWS job store tests fail sporadically with 400 error on inlined files

AWS job store tests fail sporadically with 400 error on inlined files



Discussion:
For example: 

http://jenkins.cgcloud.info/job/toil-pull-requests/572/

which is a build for PR #501.

Boto for some reason uses GET instead of POST for put_attributes. This means that the GET request size limit hits us first, before we hit the 256 \* 1024 limit for attributes, explaining why I was unable to use 256 attributes, but only 228, a limit that I determined experimentally. The inlining tests use random data which is then base64-encoded resulting in a random number of characters that have to be URL-encoded, thereby randomly breaking the URL size limit.

Unfortunately, the GET is hard-coded in Boto so we'll have to resort to monkey-patching.

For example: 

http://jenkins.cgcloud.info/job/toil-pull-requests/572/

which is a build for PR #501.

Boto for some reason uses GET instead of POST for put_attributes. This means that the GET request size limit hits us first, before we hit the 256 \* 1024 limit for attributes, explaining why I was unable to use 256 attributes, but only 228, a limit that I determined experimentally. The inlining tests use random data which is then base64-encoded resulting in a random number of characters that have to be URL-encoded, thereby randomly breaking the URL size limit.

Unfortunately, the GET is hard-coded in Boto so we'll have to resort to monkey-patching.


Resolving patch:
diff --git a/src/toil/jobStores/aws/jobStore.py b/src/toil/jobStores/aws/jobStore.py
--- a/src/toil/jobStores/aws/jobStore.py
+++ b/src/toil/jobStores/aws/jobStore.py
@@ -40,7 +40,8 @@
 from toil.jobStores.abstractJobStore import (AbstractJobStore, NoSuchJobException,
                                              ConcurrentFileModificationException,
                                              NoSuchFileException)
-from toil.jobStores.aws.sdbUtils import (SDBHelper, retry_sdb, no_such_domain, sdb_unavailable)
+from toil.jobStores.aws.sdbUtils import (SDBHelper, retry_sdb, no_such_domain, sdb_unavailable,
+                                         monkeyPatchSdbConnection)
 from toil.jobWrapper import JobWrapper
 from toil.lib.encryption import encryptionOverhead, encrypt, decrypt
 
@@ -346,6 +347,7 @@ def _connectSimpleDB(self):
             raise ValueError("Could not connect to SimpleDB. Make sure '%s' is a valid SimpleDB "
                              "region." % self.region)
         assert db is not None
+        monkeyPatchSdbConnection(db)
         return db
 
     def _connectS3(self):
@@ -534,14 +536,15 @@ def fromItem(cls, item):
             # Strings come back from SDB as unicode
             def strOrNone(s):
                 return s if s is None else str(s)
+
             # ownerID and encrypted are the only mandatory attributes
-            ownerID = strOrNone( item.get('ownerID') )
+            ownerID = strOrNone(item.get('ownerID'))
             encrypted = item.get('encrypted')
             if ownerID is None:
                 assert encrypted is None
                 return None
             else:
-                version = strOrNone( item['version'] )
+                version = strOrNone(item['version'])
                 encrypted = strict_bool(encrypted)
                 content, numContentChunks = cls.attributesToBinary(item)
                 if encrypted and content is not None:
diff --git a/src/toil/jobStores/aws/sdbUtils.py b/src/toil/jobStores/aws/sdbUtils.py
--- a/src/toil/jobStores/aws/sdbUtils.py
+++ b/src/toil/jobStores/aws/sdbUtils.py
@@ -17,6 +17,7 @@
 import bz2
 from contextlib import contextmanager
 import logging
+import types
 from boto.exception import SDBResponseError, BotoServerError
 import time
 
@@ -67,11 +68,7 @@ class SDBHelper(object):
     # requests to fail signature verification, resulting in a 403. We therefore have to
     # base64-encode values ourselves even if that means we loose a quarter of capacity.
 
-    # BEWARE: http://docs.aws.amazon.com/AmazonSimpleDB/latest/DeveloperGuide/SDBLimits.html
-    # says that we can have 256 attributes per item but found that to be inaccurate. I did a
-    # binary search and determined that it is actually only 228.
-
-    maxAttributesPerItem = 228
+    maxAttributesPerItem = 256
     maxValueSize = 1024
     maxRawValueSize = maxValueSize * 3 / 4
     # Just make sure we don't have a problem with padding or integer truncation:
@@ -147,6 +144,33 @@ def attributesToBinary(cls, attributes):
             binary = None
         return binary, numChunks
 
+from boto.sdb.connection import SDBConnection
+
+def _put_attributes_using_post(self, domain_or_name, item_name, attributes,
+                              replace=True, expected_value=None):
+    """
+    Monkey-patched version of SDBConnection.put_attributes that uses POST instead of GET
+
+    The GET version is subject to the URL length limit which kicks in before the 256 x 1024 limit
+    for attribute values. Using POST prevents that.
+
+    https://github.com/BD2KGenomics/toil/issues/502
+    """
+    domain, domain_name = self.get_domain_and_name(domain_or_name)
+    params = {'DomainName': domain_name,
+              'ItemName': item_name}
+    self._build_name_value_list(params, attributes, replace)
+    if expected_value:
+        self._build_expected_value(params, expected_value)
+    # The addition of the verb keyword argument is the only difference to put_attributes (Hannes)
+    return self.get_status('PutAttributes', params, verb='POST')
+
+def monkeyPatchSdbConnection(sdb):
+    """
+    :type sdb: SDBConnection
+    """
+    sdb.put_attributes = types.MethodType(_put_attributes_using_post, sdb)
+
 
 # FIXME: This was lifted from cgcloud-lib where we use it for EC2 retries. The only difference
 # FIXME: ... between that code and this is the name of the exception.


Test patch: