Repository: ipython/ipython

Issue:
Add support for creation of parallel task when no engine is running
IPython-0.11 complains when I try to create a parallel task when no engines are running. This makes sense for users who need results directly after they create a task. But there are also users who would like to create tasks in advance, before any engines are started. These tasks could have been stored in ipcontroller/backend database. When the first engine connects to ipcontroller it could start processing those tasks.



Discussion:
This is already true in trunk for load-balanced tasks.  Obviously, you can't send jobs to particular engines that don't exist.

Thanks. This works with the 0.12 branch.

With IPython 0.12, I just noticed that I can't get information about tasks using rc.query_db() which where created when no engine was running until I start an engine. The start of the first engine seems to trigger ipcontroller to make that information available.

Is this intended?
I would assume that all tasks which haven't been queued to an engine would have a 'pending' status.

Hmm.

The TaskScheduler disables the on_recv callback from clients when there are no engines, because it is certain that tasks cannot be submitted.  This means that they sit in the ØMQ queue, and do not trigger the callback that notifies the Hub, putting the request in the database.  I imagine I can adjust it in such a way that the tasks are pulled into Python, instead of left in the buffer.

If you need further proof let me know. Can you reproduce this behaviour?

Sorry if I wasn't clear, I was trying to explain why you see what you are seeing.  The Hub (the process that maintains the DB) will not get tasks submitted while there are no engines, because the TaskScheduler never calls recv, leaving messages in the upstream ØMQ buffer.  Essentially the queue processing is paused slightly further upstream than you would like.  I will look into addressing this.

Thank you for your explaination. Is there any workaround for this?

Modifying IPython/parallel/controller/scheduler.py made it work, but this is obviously not the solution:

```
--- ../ipython-git/IPython/parallel/controller/scheduler.py 2011-12-04 21:36:57.000000000 +0100
+++ IPython/parallel/controller/scheduler.py    2012-02-08 20:35:49.000000000 +0100
@@ -181,6 +181,7 @@

     def start(self):
         self.engine_stream.on_recv(self.dispatch_result, copy=False)
+        self.client_stream.on_recv(self.dispatch_submission, copy=False)
         self._notification_handlers = dict(
             registration_notification = self._register_engine,
             unregistration_notification = self._unregister_engine
@@ -192,12 +193,12 @@

     def resume_receiving(self):
         """Resume accepting jobs."""
-        self.client_stream.on_recv(self.dispatch_submission, copy=False)
+        #self.client_stream.on_recv(self.dispatch_submission, copy=False)

     def stop_receiving(self):
         """Stop accepting jobs while there are no engines.
         Leave them in the ZMQ queue."""
-        self.client_stream.on_recv(None)
+        #self.client_stream.on_recv(None)

     #-----------------------------------------------------------------------
     # [Un]Registration Handling
```

I think that is approximately the solution you are looking for (I'll do a PR shortly).  There certainly isn't a workaround with code as it is, because it is functioning exactly as designed - the Scheduler is entirely halted while no engines are registered.  But with the changes to the scheduler supporting dependencies, etc., I think this should work fine with only minor tweaks.


Resolving patch:
diff --git a/IPython/parallel/client/client.py b/IPython/parallel/client/client.py
--- a/IPython/parallel/client/client.py
+++ b/IPython/parallel/client/client.py
@@ -1308,7 +1308,11 @@ def queue_status(self, targets='all', verbose=False):
         verbose : bool
                 Whether to return lengths only, or lists of ids for each element
         """
-        engine_ids = self._build_targets(targets)[1]
+        if targets == 'all':
+            # allow 'all' to be evaluated on the engine
+            engine_ids = None
+        else:
+            engine_ids = self._build_targets(targets)[1]
         content = dict(targets=engine_ids, verbose=verbose)
         self.session.send(self._query_socket, "queue_request", content=content)
         idents,msg = self.session.recv(self._query_socket, 0)
diff --git a/IPython/parallel/controller/hub.py b/IPython/parallel/controller/hub.py
--- a/IPython/parallel/controller/hub.py
+++ b/IPython/parallel/controller/hub.py
@@ -430,7 +430,7 @@ def _validate_targets(self, targets):
         """turn any valid targets argument into a list of integer ids"""
         if targets is None:
             # default to all
-            targets = self.ids
+            return self.ids
 
         if isinstance(targets, (int,str,unicode)):
             # only one target specified
@@ -457,7 +457,7 @@ def _validate_targets(self, targets):
     def dispatch_monitor_traffic(self, msg):
         """all ME and Task queue messages come through here, as well as
         IOPub traffic."""
-        self.log.debug("monitor traffic: %r", msg[:2])
+        self.log.debug("monitor traffic: %r", msg[0])
         switch = msg[0]
         try:
             idents, msg = self.session.feed_identities(msg[1:])
@@ -1271,17 +1271,17 @@ def db_query(self, client_id, msg):
                 buffer_lens = [] if 'buffers' in keys else None
                 result_buffer_lens = [] if 'result_buffers' in keys else None
             else:
-                buffer_lens = []
-                result_buffer_lens = []
+                buffer_lens = None
+                result_buffer_lens = None
 
             for rec in records:
                 # buffers may be None, so double check
+                b = rec.pop('buffers', empty) or empty
                 if buffer_lens is not None:
-                    b = rec.pop('buffers', empty) or empty
                     buffer_lens.append(len(b))
                     buffers.extend(b)
+                rb = rec.pop('result_buffers', empty) or empty
                 if result_buffer_lens is not None:
-                    rb = rec.pop('result_buffers', empty) or empty
                     result_buffer_lens.append(len(rb))
                     buffers.extend(rb)
             content = dict(status='ok', records=records, buffer_lens=buffer_lens,
diff --git a/IPython/parallel/controller/scheduler.py b/IPython/parallel/controller/scheduler.py
--- a/IPython/parallel/controller/scheduler.py
+++ b/IPython/parallel/controller/scheduler.py
@@ -191,6 +191,8 @@ def _ident_default(self):
 
     def start(self):
         self.engine_stream.on_recv(self.dispatch_result, copy=False)
+        self.client_stream.on_recv(self.dispatch_submission, copy=False)
+
         self._notification_handlers = dict(
             registration_notification = self._register_engine,
             unregistration_notification = self._unregister_engine
@@ -247,8 +249,7 @@ def _register_engine(self, uid):
         self.completed[uid] = set()
         self.failed[uid] = set()
         self.pending[uid] = {}
-        if len(self.targets) == 1:
-            self.resume_receiving()
+
         # rescan the graph:
         self.update_graph(None)
 
@@ -256,7 +257,7 @@ def _unregister_engine(self, uid):
         """Existing engine with ident `uid` became unavailable."""
         if len(self.targets) == 1:
             # this was our only engine
-            self.stop_receiving()
+            pass
 
         # handle any potentially finished tasks:
         self.engine_stream.flush()
@@ -439,6 +440,11 @@ def fail_unreachable(self, msg_id, why=error.ImpossibleDependency):
 
     def maybe_run(self, msg_id, raw_msg, targets, after, follow, timeout):
         """check location dependencies, and run if they are met."""
+        self.log.debug("Attempting to assign task %s", msg_id)
+        if not self.targets:
+            # no engines, definitely can't run
+            return False
+        
         blacklist = self.blacklist.setdefault(msg_id, set())
         if follow or targets or blacklist or self.hwm:
             # we need a can_run filter


Test patch:
diff --git a/IPython/parallel/tests/test_client.py b/IPython/parallel/tests/test_client.py
--- a/IPython/parallel/tests/test_client.py
+++ b/IPython/parallel/tests/test_client.py
@@ -223,6 +223,14 @@ def test_db_query_keys(self):
         for rec in found:
             self.assertEquals(set(rec.keys()), set(['msg_id', 'submitted', 'completed']))
     
+    def test_db_query_default_keys(self):
+        """default db_query excludes buffers"""
+        found = self.client.db_query({'msg_id': {'$ne' : ''}})
+        for rec in found:
+            keys = set(rec.keys())
+            self.assertFalse('buffers' in keys, "'buffers' should not be in: %s" % keys)
+            self.assertFalse('result_buffers' in keys, "'result_buffers' should not be in: %s" % keys)
+    
     def test_db_query_msg_id(self):
         """ensure msg_id is always in db queries"""
         found = self.client.db_query({'msg_id': {'$ne' : ''}},keys=['submitted', 'completed'])