Repository: buildbot/buildbot

Issue:
convert all services to BuildbotService
This ticket is a migrated Trac ticket [3084](http://trac.buildbot.net/ticket/3084)

People contributed to the original ticket: @unknown_contributor, @djmitche, `benoit.allard@...`, @tardyp
Ticket created on: `Dec 01 2014`
Ticket last modified on: `Dec 05 2014`

---

We have a bunch of services in buildbot each of them are deriving from different base
* twisted.application.service.Service
* twisted.application.service.[[MultiService]]
* buildbot.util.service.[[AsyncService]]
* buildbot.util.service.[[AsyncMultiService]]
* buildbot.util.service.[[ReconfigurableServiceMixin]]
* buildbot.util.service.[[ClusteredService]]

Goal of this meta task is to convert all of our services to [[BuildbotService]] or [[ClusteredBuildbotService]].
This will make the code easier to understand and maintain, all our services will be
- async
- reconfigurable
- (multi) potentially a parent

The most tricky thing to do there is to move from reconfigService(new_config) which takes [[ConfigObject]] in argument, and thus configured service must know where it is configured, to a reconfigService(*args, **kwargs), which takes the constructor argument the service was configured.

Some pattern have been written in [[BuildbotService]], to handle the problem for user defined custom services, but this may not fit all the internal usecases (in buildbot everything is configured services)

Please don't resolve this bug directly, but create subtasks


---

_Comment from_: @Ben
_Date_: `Dec 01 2014`

Would this be a step toward remote reconfig of services ?

---
_Comment from_: @tardyp
_Date_: `Dec 01 2014`

>Would this be a step toward remote reconfig of services ?
This is not really the goal, but unifying the way of reconfiguring will indeed make thing less impossible.

The most complicated stuff for your request would be to send the python code for custom steps, and their dependancies.

If one does not use them, or synchronise them by other means, we could eventually think of serializing the config object, and send that through the wire.

---
_Comment from_: @Ben
_Date_: `Dec 01 2014`

I know this is not the goal, I was just wondering ...

I'm still having in mind my architecture where one single 'arbiter' / 'supervisor' / 'you-name-it' would have the config, and would care about dispatching it to masters. In this case, no need to serialise / duplicate the config, just compile it once, and dispatch it to the right instances. config are code, but once compiled, they're pretty much static, which make that whole 'throwing over the wire' easier, or am I missing something ?

---
_Comment from_: @unknown_contributor
_Date_: `Dec 01 2014`

In [changeset:"d567123cb53617d82cd22c40cfde6a4f4d3badd9"]:
```[[CommitTicketReference]] repository="" revision="d567123cb53617d82cd22c40cfde6a4f4d3badd9"
Merge pull request #1394 from tardyp/services

rename reconfigService() to reconfig[[ServiceWithBuildbotConfig]]

See ticket:3084
```

---
_Comment from_: @dustin
_Date_: `Dec 04 2014`

Ben, now that I think about it we could probably use some custom import hook magic to achieve that one-config thing you describe.  And I think it could be done in the configuration file itself, not in Buildbot.  We'd have to use some synchronous mechanism (not Twisted) to fetch Python code, but that's not hard - python-requests could probably do it.  Anyway, can you open up a bug?

---
_Comment from_: @Ben
_Date_: `Dec 05 2014`

Dustin, good idea, see #3093.




Discussion:

Resolving patch:
diff --git a/master/buildbot/worker/ec2.py b/master/buildbot/worker/ec2.py
--- a/master/buildbot/worker/ec2.py
+++ b/master/buildbot/worker/ec2.py
@@ -61,8 +61,8 @@ def __init__(self, name, password, instance_type, ami=None,
                  keypair_name=None,
                  security_name=None,
                  spot_instance=False, max_spot_price=1.6, volumes=None,
-                 placement=None, price_multiplier=1.2, tags=None, retry=1,
-                 retry_price_adjustment=1, product_description='Linux/UNIX',
+                 placement=None, price_multiplier=1.2, tags=None,
+                 product_description='Linux/UNIX',
                  subnet_id=None, security_group_ids=None, instance_profile_name=None,
                  block_device_map=None, session=None,
                  **kwargs):
@@ -117,6 +117,9 @@ def __init__(self, name, password, instance_type, ami=None,
             else:
                 # verify that regex will compile
                 re.compile(valid_ami_location_regex)
+        if spot_instance and price_multiplier is None and max_spot_price is None:
+            raise ValueError('You must provide either one, or both, of '
+                             'price_multiplier or max_spot_price')
         self.valid_ami_owners = None
         if valid_ami_owners:
             self.valid_ami_owners = [str(o) for o in valid_ami_owners]
@@ -129,9 +132,6 @@ def __init__(self, name, password, instance_type, ami=None,
         self.max_spot_price = max_spot_price
         self.volumes = volumes
         self.price_multiplier = price_multiplier
-        self.retry_price_adjustment = retry_price_adjustment
-        self.retry = retry
-        self.attempt = 1
         self.product_description = product_description
 
         if None not in [placement, region]:
@@ -451,7 +451,7 @@ def _stop_instance(self, instance, fast):
                 (self.__class__.__name__, self.workername,
                  instance.id, goal, duration // 60, duration % 60))
 
-    def _submit_request(self):
+    def _bid_price_from_spot_price_history(self):
         timestamp_yesterday = time.gmtime(int(time.time() - 86400))
         spot_history_starttime = time.strftime(
             '%Y-%m-%dT%H:%M:%SZ', timestamp_yesterday)
@@ -466,25 +466,23 @@ def _submit_request(self):
                 price_sum += float(price['SpotPrice'])
                 price_count += 1
         if price_count == 0:
-            self.current_spot_price = 0.02
+            bid_price = 0.02
         else:
-            self.current_spot_price = (price_sum / price_count) * self.price_multiplier
-        if self.current_spot_price > self.max_spot_price:
-            log.msg('%s %s calculated spot price %0.3f exceeds '
-                    'configured maximum of %0.3f' %
-                    (self.__class__.__name__, self.workername,
-                     self.current_spot_price, self.max_spot_price))
-            raise LatentWorkerFailedToSubstantiate()
+            bid_price = (price_sum / price_count) * self.price_multiplier
+        return bid_price
+
+    def _request_spot_instance(self):
+        if self.price_multiplier is None:
+            bid_price = self.max_spot_price
         else:
-            if self.retry > 1:
-                log.msg('%s %s requesting spot instance with price %0.4f, attempt %d of %d' %
-                        (self.__class__.__name__, self.workername, self.current_spot_price, self.attempt,
-                         self.retry))
-            else:
-                log.msg('%s %s requesting spot instance with price %0.4f' %
-                        (self.__class__.__name__, self.workername, self.current_spot_price))
+            bid_price = self._bid_price_from_spot_price_history()
+            if self.max_spot_price is not None \
+               and bid_price > self.max_spot_price:
+                bid_price = self.max_spot_price
+        log.msg('%s %s requesting spot instance with price %0.4f' %
+                (self.__class__.__name__, self.workername, bid_price))
         reservations = self.ec2.meta.client.request_spot_instances(
-                SpotPrice=str(self.current_spot_price),
+                SpotPrice=str(bid_price),
                 LaunchSpecification=self._remove_none_opts(
                     ImageId=self.ami,
                     KeyName=self.keypair_name,
@@ -504,31 +502,12 @@ def _submit_request(self):
                 )
         request, success = self._wait_for_request(reservations['SpotInstanceRequests'][0])
         if not success:
-            return request, None, None, False
-        else:
-            instance_id = request['InstanceId']
-            self.instance = self.ec2.Instance(instance_id)
-            image = self.get_image()
-            instance_id, start_time = self._wait_for_instance()
-            return instance_id, image.id, start_time, True
-
-    def _request_spot_instance(self):
-        if self.retry > 1:
-            for attempt in range(1, self.retry + 1):
-                self.attempt = attempt
-                instance_id, image_id, start_time, success = self._submit_request()
-                if success:
-                    break
-                if attempt >= self.retry:
-                    self.attempt = 0
-                    log.msg('%s %s failed to substantiate after %d requests' %
-                            (self.__class__.__name__, self.workername, self.retry))
-                    raise LatentWorkerFailedToSubstantiate()
-        else:
-            instance_id, image_id, start_time, success = self._submit_request()
-            if not success:
-                raise LatentWorkerFailedToSubstantiate()
-        return instance_id, image_id, start_time
+            raise LatentWorkerFailedToSubstantiate()
+        instance_id = request['InstanceId']
+        self.instance = self.ec2.Instance(instance_id)
+        image = self.get_image()
+        instance_id, start_time = self._wait_for_instance()
+        return instance_id, image.id, start_time
 
     def _wait_for_instance(self):
         log.msg('%s %s waiting for instance %s to start' %
@@ -594,8 +573,8 @@ def _wait_for_request(self, reservation):
                 SpotInstanceRequestIds=[request['SpotInstanceRequestId']])
             log.msg('%s %s spot request rejected, spot price too low' %
                     (self.__class__.__name__, self.workername))
-            self.current_spot_price *= self.retry_price_adjustment
-            return request, False
+            raise LatentWorkerFailedToSubstantiate(
+                request['SpotInstanceRequestId'], request.status)
         else:
             log.msg('%s %s failed to fulfill spot request %s with status %s' %
                     (self.__class__.__name__, self.workername,


Test patch:
diff --git a/master/buildbot/test/unit/test_worker_ec2.py b/master/buildbot/test/unit/test_worker_ec2.py
--- a/master/buildbot/test/unit/test_worker_ec2.py
+++ b/master/buildbot/test/unit/test_worker_ec2.py
@@ -435,43 +435,11 @@ def test_start_spot_instance(self):
             Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
         instances = list(instances)
         self.assertTrue(bs.spot_instance)
-        self.assertEqual(bs.retry, 1)
         self.assertEqual(bs.product_description, product_description)
         self.assertEqual(len(instances), 1)
         self.assertEqual(instances[0].id, instance_id)
         self.assertEqual(instances[0].tags, [])
 
-    @mock_ec2
-    def test_start_spot_instance_retry(self):
-        c, r = self.botoSetup('latent_buildbot_slave')
-        amis = list(r.images.all())
-        product_description = 'Linux/Unix'
-        retry = 3
-        with assertProducesWarnings(
-                DeprecatedWorkerNameWarning,
-                messages_patterns=[
-                    r"Use of default value of 'keypair_name' of "
-                    r"EC2LatentWorker constructor is deprecated",
-                    r"Use of default value of 'security_name' of "
-                    r"EC2LatentWorker constructor is deprecated"
-                ]):
-            bs = ec2.EC2LatentWorker('bot1', 'sekrit', 'm1.large',
-                                     identifier='publickey',
-                                     secret_identifier='privatekey',
-                                     ami=amis[0].id, retry=retry,
-                                     spot_instance=True, max_spot_price=1.5,
-                                     product_description=product_description
-                                     )
-        id, _, _ = bs._start_instance()
-        instances = r.instances.filter(
-            Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
-        instances = list(instances)
-        self.assertTrue(bs.spot_instance)
-        self.assertEqual(bs.retry, 3)
-        self.assertEqual(bs.attempt, 1)
-        self.assertEqual(len(instances), 1)
-        self.assertEqual(instances[0].id, id)
-
     @mock_ec2
     def test_get_image_ami(self):
         c, r = self.botoSetup('latent_buildbot_slave')
@@ -534,15 +502,26 @@ def create_worker():
         self.assertRaises(ValueError, create_worker)
 
     @mock_ec2
-    def test_start_spot_instance_retry_low_price(self):
+    def test_fail_multiplier_and_max_are_none(self):
         '''
-        This test should attempt to start an instance that will be rejected with
-        price-too-low. At this point, the ec2 worker code should increment
-        bs.attempt and multiply the price by bs.retry_price_adjustment. This
-        should continue for bs.retry iterations or until the spot request is
-        accepted.
+        price_multiplier and max_spot_price may not be None at the same time.
         '''
-        raise unittest.SkipTest("Requires un-released functionality in moto.")
+        c, r = self.botoSetup()
+        amis = list(r.images.all())
+
+        def create_worker():
+            ec2.EC2LatentWorker('bot1', 'sekrit', 'm1.large',
+                                identifier='publickey',
+                                secret_identifier='privatekey',
+                                keypair_name="latent_buildbot_worker",
+                                security_name='latent_buildbot_worker',
+                                ami=amis[0].id,
+                                region='us-west-1',
+                                spot_instance=True,
+                                price_multiplier=None,
+                                max_spot_price=None
+                                )
+        self.assertRaises(ValueError, create_worker)
 
 
 class TestEC2LatentWorkerDefaultKeyairSecurityGroup(unittest.TestCase):