Repository: Parsl/parsl

Issue:
Add block ID to try table
**Is your feature request related to a problem? Please describe.**

The monitoring database should record which block (and worker, ...) a particular try happened on.

This is useful for understanding where tasks run at a finer grain than the hostname, which can be re-used in multiple blocks over the course of  a run. 

**Additional context**

This is for lsst desc drp

cc @tomglanzman


Discussion:
Looks like it should not be too difficult to add. Do you expect this to come in 1.1 release?
The DRP workflow runs off a branch that regularly has master merged, so it doesn't need to have any main release target.
Hi @ZhuozhaoLi , just checking to see if there is any progress on this?  Having it in Ben's lsst-desc branch would enable me to move ahead with work defining standard db queries.  Thanks!
@TomGlanzman I will come back to this in a week or so.

Resolving patch:
diff --git a/parsl/executors/high_throughput/process_worker_pool.py b/parsl/executors/high_throughput/process_worker_pool.py
--- a/parsl/executors/high_throughput/process_worker_pool.py
+++ b/parsl/executors/high_throughput/process_worker_pool.py
@@ -497,6 +497,7 @@ def worker(worker_id, pool_id, pool_size, task_queue, result_queue, worker_queue
     os.environ['PARSL_WORKER_RANK'] = str(worker_id)
     os.environ['PARSL_WORKER_COUNT'] = str(pool_size)
     os.environ['PARSL_WORKER_POOL_ID'] = str(pool_id)
+    os.environ['PARSL_WORKER_BLOCK_ID'] = str(args.block_id)
 
     # Sync worker with master
     logger.info('Worker {} started'.format(worker_id))
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -156,6 +156,7 @@ class Try(Base):
         task_id = Column('task_id', Integer, nullable=False)
         run_id = Column('run_id', Text, nullable=False)
 
+        block_id = Column('block_id', Text, nullable=True)
         hostname = Column('hostname', Text, nullable=True)
 
         task_executor = Column('task_executor', Text, nullable=False)
@@ -504,7 +505,7 @@ def start(self,
                     self._update(table=TRY,
                                  columns=['task_try_time_running',
                                           'run_id', 'task_id', 'try_id',
-                                          'hostname'],
+                                          'block_id', 'hostname'],
                                  messages=reprocessable_first_resource_messages)
             except Exception:
                 logger.exception("Exception in db loop: this might have been a malformed message, or some other error. monitoring data may have been lost")
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -548,6 +548,7 @@ def send_first_message(try_id: int,
                        monitoring_hub_url: str,
                        run_id: str) -> None:
     import platform
+    import os
 
     radio = UDPRadio(monitoring_hub_url,
                      source_id=task_id)
@@ -556,6 +557,7 @@ def send_first_message(try_id: int,
            'try_id': try_id,
            'task_id': task_id,
            'hostname': platform.node(),
+           'block_id': os.environ.get('PARSL_WORKER_BLOCK_ID'),
            'first_msg': True,
            'timestamp': datetime.datetime.now()
     }


Test patch: