Repository: deis/deis

Issue:
'Node bootstrap error' when scaling nodes
```
(deis)rhialto:example-go matt$ deis formations:create testformation --flavor=ec2-us-west-2
Creating formation... done, created testformation

Creating runtime layer... done in 2s

Use `deis nodes:scale testformation runtime=1` to scale a basic formation
(deis)rhialto:example-go matt$ deis nodes:scale testformation runtime=1
Scaling nodes... but first, coffee!
500 INTERNAL SERVER ERROR
<h1>Server Error (500)</h1>
```

On the server:

``` console
==> /var/log/deis/celeryd.log <==
[2013-09-30 11:10:39,752: ERROR/MainProcess] Task api.tasks.build_node[001666ca-f329-4794-a412-429785394b8b] raised exception: RuntimeError(u'Node Bootstrap Error',)
Traceback (most recent call last):
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/task/trace.py", line 233, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/task/trace.py", line 420, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/deis/controller/api/tasks.py", line 36, in build_node
    CM.bootstrap_node(node.flat())
  File "/opt/deis/controller/cm/chef.py", line 107, in bootstrap_node
    raise RuntimeError('Node Bootstrap Error')
None: Node Bootstrap Error
```

'Node bootstrap error' when scaling nodes
```
(deis)rhialto:example-go matt$ deis formations:create testformation --flavor=ec2-us-west-2
Creating formation... done, created testformation

Creating runtime layer... done in 2s

Use `deis nodes:scale testformation runtime=1` to scale a basic formation
(deis)rhialto:example-go matt$ deis nodes:scale testformation runtime=1
Scaling nodes... but first, coffee!
500 INTERNAL SERVER ERROR
<h1>Server Error (500)</h1>
```

On the server:

``` console
==> /var/log/deis/celeryd.log <==
[2013-09-30 11:10:39,752: ERROR/MainProcess] Task api.tasks.build_node[001666ca-f329-4794-a412-429785394b8b] raised exception: RuntimeError(u'Node Bootstrap Error',)
Traceback (most recent call last):
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/task/trace.py", line 233, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/deis/controller/venv/local/lib/python2.7/site-packages/celery/task/trace.py", line 420, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/deis/controller/api/tasks.py", line 36, in build_node
    CM.bootstrap_node(node.flat())
  File "/opt/deis/controller/cm/chef.py", line 107, in bootstrap_node
    raise RuntimeError('Node Bootstrap Error')
None: Node Bootstrap Error
```



Discussion:
Seems like "empty run list" is the cause?

``````
~/controller$ knife bootstrap ec2-1-2-3-4.compute-1.amazonaws.com --identity-file /tmp/tmpzlONdB --node-name testformation-runtime-1 --sudo --ssh-user ubuntu --ssh-port 22 --bootstrap-version 11.4.4 --no-host-key-verify --run-list recipe[deis],recipe[deis::runtime],recipe[deis::proxy]
Bootstrapping Chef on ec2-1-2-3-4.compute-1.amazonaws.com
ec2-1-2-3-4.compute-1.amazonaws.com Starting Chef Client, version 11.4.4
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com resolving cookbooks for run list: ["deis", "deis::runtime", "deis::proxy"]
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Synchronizing Cookbooks:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - deis
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - apt
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - sudo
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - rsyslog
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Compiling Cookbooks...
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ================================================================================
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Recipe Compile Error in /var/chef/cache/cookbooks/deis/recipes/runtime.rb
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ================================================================================
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com NameError
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ---------
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Cannot find a resource for syslog_servers on ubuntu version 12.04
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Cookbook Trace:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ---------------
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   /var/chef/cache/cookbooks/rsyslog/recipes/client.rb:34:in `block in from_file'
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   /var/chef/cache/cookbooks/rsyslog/recipes/client.rb:33:in `from_file'
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   /var/chef/cache/cookbooks/deis/recipes/runtime.rb:1:in `from_file'
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Relevant File Content:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ----------------------
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com /var/chef/cache/cookbooks/rsyslog/recipes/client.rb:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  27:    if node['rsyslog']['server_ip']
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  28:      # handle node['rsyslog']['server_ip'] being a string or an array
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  29:      rsyslog_servers = Array(node['rsyslog']['server_ip'])
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  30:    end
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  31:  
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  32:    # add all syslog servers to the syslog_servers array
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  33:    search(:node, node["rsyslog"]["server_search"]) do |result|
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  34>>     syslog_servers << result['ipaddress']
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  35:    end
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  36:  
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  37:    if rsyslog_servers.empty?
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  38:      Chef::Application.fatal!("The rsyslog::client recipe was unable to determine the remote syslog server. Checked both the server_ip attribute and search()")
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  39:    end
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  40:  
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  41:    template "/etc/rsyslog.d/49-remote.conf" do
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  42:      only_if { node['rsyslog']['remote_logs'] && !rsyslog_servers.empty? }
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  43:      source "49-remote.conf.erb"
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] ERROR: Running exception handlers
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json```
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] ERROR: Exception handlers complete
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Chef Client failed. 0 resources updated
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] FATAL: NameError: Cannot find a resource for syslog_servers on ubuntu version 12.04
ec2-1-2-3-4.compute-1.amazonaws.com 
``````

I am unable to reproduce this with updated Gems and Cookbooks.  I'll try a fresh controller next.

Seems like "empty run list" is the cause?

``````
~/controller$ knife bootstrap ec2-1-2-3-4.compute-1.amazonaws.com --identity-file /tmp/tmpzlONdB --node-name testformation-runtime-1 --sudo --ssh-user ubuntu --ssh-port 22 --bootstrap-version 11.4.4 --no-host-key-verify --run-list recipe[deis],recipe[deis::runtime],recipe[deis::proxy]
Bootstrapping Chef on ec2-1-2-3-4.compute-1.amazonaws.com
ec2-1-2-3-4.compute-1.amazonaws.com Starting Chef Client, version 11.4.4
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com resolving cookbooks for run list: ["deis", "deis::runtime", "deis::proxy"]
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Synchronizing Cookbooks:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - deis
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - apt
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - sudo
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   - rsyslog
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Compiling Cookbooks...
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ================================================================================
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Recipe Compile Error in /var/chef/cache/cookbooks/deis/recipes/runtime.rb
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ================================================================================
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com NameError
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ---------
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Cannot find a resource for syslog_servers on ubuntu version 12.04
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Cookbook Trace:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ---------------
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   /var/chef/cache/cookbooks/rsyslog/recipes/client.rb:34:in `block in from_file'
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   /var/chef/cache/cookbooks/rsyslog/recipes/client.rb:33:in `from_file'
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com   /var/chef/cache/cookbooks/deis/recipes/runtime.rb:1:in `from_file'
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Relevant File Content:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com ----------------------
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com /var/chef/cache/cookbooks/rsyslog/recipes/client.rb:
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  27:    if node['rsyslog']['server_ip']
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  28:      # handle node['rsyslog']['server_ip'] being a string or an array
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  29:      rsyslog_servers = Array(node['rsyslog']['server_ip'])
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  30:    end
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  31:  
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  32:    # add all syslog servers to the syslog_servers array
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  33:    search(:node, node["rsyslog"]["server_search"]) do |result|
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  34>>     syslog_servers << result['ipaddress']
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  35:    end
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  36:  
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  37:    if rsyslog_servers.empty?
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  38:      Chef::Application.fatal!("The rsyslog::client recipe was unable to determine the remote syslog server. Checked both the server_ip attribute and search()")
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  39:    end
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  40:  
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  41:    template "/etc/rsyslog.d/49-remote.conf" do
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  42:      only_if { node['rsyslog']['remote_logs'] && !rsyslog_servers.empty? }
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com  43:      source "49-remote.conf.erb"
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] ERROR: Running exception handlers
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json```
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] ERROR: Exception handlers complete
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com Chef Client failed. 0 resources updated
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
ec2-1-2-3-4.compute-1.amazonaws.com 
ec2-1-2-3-4.compute-1.amazonaws.com [2013-09-30T19:20:31+00:00] FATAL: NameError: Cannot find a resource for syslog_servers on ubuntu version 12.04
ec2-1-2-3-4.compute-1.amazonaws.com 
``````

I am unable to reproduce this with updated Gems and Cookbooks.  I'll try a fresh controller next.


Resolving patch:
diff --git a/cm/chef.py b/cm/chef.py
--- a/cm/chef.py
+++ b/cm/chef.py
@@ -84,6 +84,7 @@ def bootstrap_node(node):
     # write out private key and prepare to `knife bootstrap`
     try:
         _, pk_path = tempfile.mkstemp()
+        _, output_path = tempfile.mkstemp()
         with open(pk_path, 'w') as f:
             f.write(node['ssh_private_key'])
         # build knife bootstrap command
@@ -96,18 +97,25 @@ def bootstrap_node(node):
         args.extend(['--no-host-key-verify'])
         args.extend(['--run-list', _construct_run_list(node)])
         print(' '.join(args))
+        # tee the command's output to a tempfile
+        args.extend(['|', 'tee', output_path])
         # TODO: figure out why home isn't being set correctly for knife exec
         env = os.environ.copy()
         env['HOME'] = '/opt/deis'
         # execute knife bootstrap
-        p = subprocess.Popen(args, env=env, stderr=subprocess.PIPE)
+        p = subprocess.Popen(' '.join(args), env=env, shell=True)
         rc = p.wait()
+        # always print knife output
+        with open(output_path) as f:
+            output = f.read()
+        print(output)
+        # raise an exception if bootstrap failed
         if rc != 0:
-            print(p.stderr.read())
             raise RuntimeError('Node Bootstrap Error')
-    # remove private key from fileystem
+    # remove temp files from filesystem
     finally:
-        pass  # os.remove(pk_path)
+        os.remove(pk_path)
+        os.remove(output_path)
 
 
 def _construct_run_list(node):


Test patch: