Repository: ICRAR/daliuge
Pull request #314: LIU-5: Provide improved FileDROP naming and support for DALiuGE visibility of PyFuncApp side-effects 
Author: login: myxie
Created: 2025-03-12T11:50:06Z | Merged: 2025-04-29T02:40:50Z

# JIRA Ticket 
<!---
If there is no JIRA ticket, please consider raising one to summarise the work there. Alternatively, link to a GitHub if that exists._
--->
LIU-5
# Type
<!---Select what type of work this PR is addressing. This is useful for preparing the [Changelog](https://github.com/ICRAR/daliuge/blob/master/CHANGELOG.md)--->

- [x] Feature (addition)
- [ ] Bug fix
- [ ] Refactor (change)
- [ ] Documentation 

# Problem/Issue 

<!--- Provide an overview of what this PR address--->

The dataflow approach used by DALiuGE assumes that the output of an AppDROP (DockerApp, PyFuncApp etc.) is managed and controlled by the daliuge-enginge. This is how DALiuGE can keep track of outputs from any AppDROP and write it to an arbitrary DataDROP, or swap between DataDROP formats on the graph. 

However, some applications (especially PyFuncApps) will perform their own writing that we have no control over in the daliuge-engine; for example, a Python function that writes a file out and has no return. Currently, DALiuGe has no way of knowing about that file, as it is a side-effect of the function. 

Typically, applications that do have side-effects will either require or provide an option to pass a filename. What would improve DALiuGE's visibility of these side-effect files is an ability to control the Filename of the side-effect and pair it with a FileDROP; this way, the engine knows about the data and can use later in the workflow.

# Solution
<!--- A description of the solution, including design decisions or compromises made. Consider adding a figure or example of how the behaviour has changed. ---> 

To fix this, I have: 

- Enabled complete multiple InputPorts to DataDROPs
- Added support to map InputOutput/OutputPorts to function parameters of a PyFuncApp drop. 
- Unittests for FileDROPs and named ports
- Restructured pyfunc.py to improve readability and isolate specific functionality.  
- Added f-string style support for filenames with keyword substitutions: 
    -  for example, passing "my_filename_{datetime}.txt" will produce "my_file_2025-04-7"
 
This leads to the following graph working: 

![image](https://github.com/user-attachments/assets/8255db30-b4c7-4794-b841-fdfb0e6cab36)


I can produce the expected files from it without any DLG output. 

![image](https://github.com/user-attachments/assets/f4e1cbc0-907b-49e7-a7a4-3ff639feb6f7)


<!--- Provide a description of documentation added, testing, including manual and programmatic ---> 

- [x] Unittests added
  - Reason for not adding unittests (remove this line if added) 
- [ ] Documentation added
    - I will add documentation for this as part of our ongoing [Documentation updates](https://icrar.atlassian.net/wiki/spaces/DALIUGE/pages/7015497730/DALiuGE+Documentation+Checklist)

## Summary by Sourcery

Improve FileDROP naming and support for input-output filepaths in DALiuGE

New Features:
- Add support for dynamic filepath generation using f-string-like replacements
- Enable mapping of InputOutput/OutputPorts to function parameters in PyFuncApp
- Implement flexible filename generation for FileDROPs

Bug Fixes:
- Fix issues with side-effect file handling in PyFuncApps
- Improve handling of function signatures and argument parsing

Enhancements:
- Restructure pyfunc.py to improve readability and isolate specific functionality
- Improve error handling and signature detection for Python functions
- Enhance port mapping and argument handling in PyFuncApp

Tests:
- Add unit tests for FileDROPs and named ports
- Implement comprehensive testing for filepath generation and input-output port mapping


Diff:
diff --git a/.github/workflows/create-palettes.yml b/.github/workflows/create-palettes.yml
index dfaf721d2..3f6c58fe2 100644
--- a/.github/workflows/create-palettes.yml
+++ b/.github/workflows/create-palettes.yml
@@ -24,7 +24,7 @@ jobs:
         run: |
           sudo apt-get update && sudo apt-get install -y doxygen xsltproc
           pip install BlockDAG
-          pip install dlg_paletteGen
+          pip install dlg-paletteGen
 
       - name: Configure git
         run: |
diff --git a/.github/workflows/run-unit-tests.yml b/.github/workflows/run-unit-tests.yml
index ad0a61fbf..e9188c355 100644
--- a/.github/workflows/run-unit-tests.yml
+++ b/.github/workflows/run-unit-tests.yml
@@ -87,7 +87,7 @@ jobs:
 
   finish:
     needs: run_tests
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-22.04
     steps:
       - name: Coveralls Finished
         uses: coverallsapp/github-action@master
diff --git a/daliuge-common/dlg/common/__init__.py b/daliuge-common/dlg/common/__init__.py
index 9118b2979..b598c8604 100644
--- a/daliuge-common/dlg/common/__init__.py
+++ b/daliuge-common/dlg/common/__init__.py
@@ -70,6 +70,44 @@ class dropdict(dict):
     with JSON -> DROP representation transformations, and the different
     repositories where graph templates are expected to be found by the
     DROPManager.
+
+    Supported Keys
+    ---------------
+    Common:
+
+        "oid": str
+        "iid": str
+        "lg_key": str
+        "name": str
+        "categoryType": str
+        "dropclass": str
+        "storage": str
+        "rank": list[int]
+        "reprodata": dict
+        "loop_ctx": Union[None/int]
+        "weight": int
+        "applicationArgs": dict
+        "constraintParams": dict
+        "componentParams": dict
+        "fields": list[dict]
+        "data_volume": str
+        "group_end": str (rep. of boolean value '0'==False)
+        "check_file_path_exists": str (rep. of boolean value '0'==False)
+
+    AppDROP only:
+
+        "outputs": list[dict]
+
+    DataDROP only:
+
+        "persist": bool
+        "producers": list[dict]
+        "port_map": dict
+
+    FileDROP only:
+
+        "filepath": str
+        "dirname": str
     """
 
     def __init__(self, init_dict=None):
diff --git a/daliuge-translator/dlg/dropmake/path_utils.py b/daliuge-common/dlg/common/path_utils.py
similarity index 100%
rename from daliuge-translator/dlg/dropmake/path_utils.py
rename to daliuge-common/dlg/common/path_utils.py
diff --git a/daliuge-engine/dlg/apps/pyfunc.py b/daliuge-engine/dlg/apps/pyfunc.py
index 73407373a..3d915e89a 100644
--- a/daliuge-engine/dlg/apps/pyfunc.py
+++ b/daliuge-engine/dlg/apps/pyfunc.py
@@ -24,7 +24,6 @@
 import ast
 import base64
 import collections
-from enum import Enum
 import importlib
 import inspect
 import json
@@ -33,19 +32,21 @@
 import pickle
 import re
 
-from typing import Callable, Union
+from typing import Callable, Union, Union
 import dill
 from io import StringIO
 from contextlib import redirect_stdout
 
 from dlg import drop_loaders
+from dlg.data.path_builder import filepath_from_string
 from dlg.utils import serialize_data, deserialize_data
 from dlg.named_port_utils import (
+    Argument,
     DropParser,
     check_ports_dict,
     get_port_reader_function,
     identify_named_ports,
-    replace_named_ports,
+    replace_named_ports
 )
 from dlg.apps.app_base import BarrierAppDROP
 from dlg.exceptions import InvalidDropException
@@ -76,12 +77,12 @@ def serialize_func(f):
     if a.defaults:
         fdefaults["kwargs"] = dict(
             zip(
-                a.args[-len(a.defaults) :],
+                a.args[-len(a.defaults):],
                 [serialize_data(d) for d in a.defaults],
             )
         )
         adefaults["kwargs"] = dict(
-            zip(a.args[-len(a.defaults) :], [d for d in a.defaults])
+            zip(a.args[-len(a.defaults):], [d for d in a.defaults])
         )
     logger.debug(f"Introspection of function {f}: {a}")
     logger.debug("Defaults for function %r: %r", f, adefaults)
@@ -268,6 +269,9 @@ class PyFuncApp(BarrierAppDROP):
     )
 
     func_name = dlg_string_param("func_name", None)
+    # func_code = dlg_bytes_param("func_code", None) # bytes or base64 string
+    input_parser: DropParser = dlg_enum_param(DropParser, "input_parser", DropParser.PICKLE)  # type: ignore
+    output_parser: DropParser = dlg_enum_param(DropParser, "output_parser", DropParser.PICKLE)  # type: ignore
     func_arg_mapping = dlg_dict_param("func_arg_mapping", {})
     func_defaults = dlg_dict_param("func_defaults", {})
     func: Callable
@@ -277,6 +281,12 @@ class PyFuncApp(BarrierAppDROP):
     input_parser = dlg_enum_param(DropParser, "input_parser", DropParser.DILL)
     output_parser = dlg_enum_param(DropParser, "output_parser", DropParser.DILL)
 
+    def __repr__(self):
+        if hasattr(self, "func"):
+            return  f"{self.__class__.__name__}_{self.func}_{self.oid}"
+        else:
+            return f"{self.__class__.__name__}"
+
     def _mixin_func_defaults(self):
         """
         func_defaults can be passed in through an argument, but this is only used for
@@ -286,17 +296,17 @@ def _mixin_func_defaults(self):
         """
         logger.debug(f"Starting evaluation of func_defaults: {self.func_defaults}")
         if (
-            isinstance(self.func_defaults, dict)
-            and len(self.func_defaults) > 0
-            and list(self.func_defaults.keys()) == ["kwargs", "args"]
+                isinstance(self.func_defaults, dict)
+                and len(self.func_defaults) > 0
+                and list(self.func_defaults.keys()) == ["kwargs", "args"]
         ):
             for arg in self.func_defaults["args"]:
                 self.func_defaults["kwargs"][arg] = arg
                 self.func_defaults = self.func_defaults["kwargs"]
         elif (
-            isinstance(self.func_defaults, dict)
-            and "kwargs" in self.func_defaults
-            and isinstance(self.func_defaults["kwargs"], dict)
+                isinstance(self.func_defaults, dict)
+                and "kwargs" in self.func_defaults
+                and isinstance(self.func_defaults["kwargs"], dict)
         ):
             self.func_defaults = self.func_defaults["kwargs"]
         # we came all this way, now assume that any resulting dict is correct
@@ -326,11 +336,6 @@ def _init_fn_defaults(self):
         Multiple options exist and some are here for compatibility.
         """
         logger.debug(f"Starting evaluation of function signature")
-        self.argsig = inspect.signature(self.func)
-        self.argnames = list(self.argsig.parameters.keys())
-        logger.debug("Function signature: %s", self.argsig)
-        args = list(self.argsig.parameters.keys())
-        self.fn_nargs = len(self.argsig.parameters)
         # set defaults
         self.fn_defaults = {}
         self.fn_nposkw = 0
@@ -342,6 +347,25 @@ def _init_fn_defaults(self):
         self.varargs = False
         self.varkw = False
         self.fn_ndef = 0
+        self.argnames = []
+        self.argsig = None
+        self.fn_nargs = 0
+        try:
+            self.argsig = inspect.signature(self.func)
+            self.argnames = list(self.argsig.parameters.keys())
+            logger.debug("Function signature: %s", self.argsig)
+            args = list(self.argsig.parameters.keys())
+            self.fn_nargs = len(self.argsig.parameters)
+            self.populate_arguments_without_signature = False
+        except ValueError:
+            logger.warning("Unable to uncover signature from function. This is likely a "
+                           "C-based Python function. No positional/keyword arguments "
+                           "determined.")
+            self.argsig = self.func
+            self.populate_arguments_without_signature = True
+
+            return False
+
         for k, p in self.argsig.parameters.items():
             if not isinstance(p, str):
                 if p.kind == p.POSITIONAL_OR_KEYWORD:
@@ -369,6 +393,7 @@ def _init_fn_defaults(self):
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
@@ -378,139 +403,250 @@ def _clean_applicationArgs(self):
             "func_code",
             "func_name",
             "func_arg_mapping",
-            "input_parser",
+                                  "input_parser",
             "output_parser",
             "func_defaults",
-            "pickle",
+           
+                                  "pickle",
         ]
 
         for kw in self.func_def_keywords:
-            if kw in self._applicationArgs:  # these are the preferred ones now
-                if isinstance(
-                    self._applicationArgs[kw]["value"],
-                    bool
-                    or self._applicationArgs[kw]["value"]
-                    or self._applicationArgs[kw]["precious"],
-                ):
-                    # only transfer if there is a value or precious is True
-                    self._applicationArgs.pop(kw)
-
-    def _initialise_positional_args(self, argsDict: dict):
+            # these are the preferred ones now
+            if kw in self._applicationArgs and isinstance(
+                self._applicationArgs[kw]["value"],
+                bool or self._applicationArgs[kw]["value"] or
+                self._applicationArgs[kw]["precious"],
+            ):
+                # only transfer if there is a value or precious is True
+                self._applicationArgs.pop(kw)
+
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
         for arg, value in argsDict.items():
-            if arg in self._applicationArgs:
-                encoding = self._applicationArgs[arg].get("encoding", self.input_parser)
-            else:
-                encoding = self.input_parser
-            argsDict[arg] = {"value": value, "encoding": encoding}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
-                argsDict[arg]["value"] = param_arg
-
+                argsDict[arg].value = param_arg
+                argsDict[arg].encoding = "dill"
         return argsDict
 
     def _get_arg_info(self, arg):
-
+        """
+        Get necessary information for populating the argument
+        """
         value = self._applicationArgs[arg]["value"]
-        encoding = self._applicationArgs[arg].get("encoding", self.input_parser)
+        encoding = self._applicationArgs[arg].get("encoding")
+        precious = self._applicationArgs[arg].get("precious")
+        positional = self._applicationArgs[arg].get("positional")
+        portType = self._applicationArgs[arg].get("usage")
 
-        return value, encoding
+        return value, encoding, precious, positional, portType
 
-    def _init_appArgs(self, pargsDict: dict, keyargsDict: dict, posargs: list) -> list:
+    def _populate_arguments_with_graph_data(self, positionalArgsMap, keywordArgsMap):
         """
-        Identify and fill application arguments.
+         Go through the parameters that were provided by the user to the Logical Graph and
+        populate the application argument with the data.
 
-        Deals with positional and keyword arguments.
+        If the user didn't provide any input, or the data is expected to come through an
+        InputPort, the app argument will not be updated.
 
-        pargsDict: dict,
+        :param positionalArgsMap:
+        :param keywordArgsMap:
 
-        keyargsDict:
+        :return: positionalArgsMap,
+        :return: keywordArgsMap
+        :return: input_outputs
+        """
 
-        posargs: list
+        vparg = []
+        vkarg = {}
+        input_outputs = []
+
+        # Port-types that we may receive a filename/path from that is used as an output
+        # reference
+        filename_ports = ["InputOutput", "OutputPort"]
+
+        # Legacy, used for graphs reliant on input/output parser
+        if not self._applicationArgs:
+            encoding = self.input_parser or DropParser.DILL
+            for key in positionalArgsMap:
+                positionalArgsMap[key].encoding = encoding
+            logger.debug("AppArgs/pargsDict: %s", positionalArgsMap)
+
+            return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
+
+        # If we overwrite any defaults with what is store in AppArgs "value"
+        for arg in self._applicationArgs:
+            # check value type and interpret
+            value, encoding, precious, positional, portType = self._get_arg_info(
+                arg)
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                try:
+                    value = ast.literal_eval(value)
+                    logger.debug(f"Evaluated %s to %s",
+                                 value, type(value))
+                    self._applicationArgs[arg].value = value
+                except ValueError:
+                    logger.error("Unable to evaluate %s",
+                                 self._applicationArgs[arg]["value"])
+            for arg_map in [positionalArgsMap, keywordArgsMap]:
+                if arg in arg_map:
+                    argument = arg_map[arg]
+                    argument.value = value or argument.value
+                    argument.encoding = encoding or argument.encoding
+                    argument.precious = precious or argument.precious
+                    argument.positional = positional or argument.positional
+                    arg_map[arg] = argument
+
+            if portType in filename_ports:
+                input_outputs.append(arg)
+
+        # Remove parameters of function that have been found in applicationArgs
+        _ = [self._applicationArgs.pop(k) for k in positionalArgsMap if
+             k in self._applicationArgs]
+        logger.debug("Updated posargs dictionary: %s", positionalArgsMap)
+        logger.debug("Updated keyargs dictionary: %s", keywordArgsMap)
+
+        # Put all remaining arguments into *args and **kwargs
+        logger.debug(f"Remaining AppArguments {self._applicationArgs}")
+        for arg in self._applicationArgs:
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                value = ast.literal_eval(self._applicationArgs[arg]["value"])
+            else:
+                value = self._applicationArgs[arg]["value"]
+            if (self._applicationArgs[arg]["positional"] or
+                    self.populate_arguments_without_signature):
+                vparg.append(value)
+            else:
+                vkarg[arg] = value
 
+        return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
 
-        Returns:
-        --------
-        list, [funcargs, pargs]
+
+    def _map_parameters_to_func_args(self, positionalArgs: dict, keywordArguments: dict) -> tuple:
+        """
+        Given the keyword and postional arguments of the function we are using, map
+        the EAGLE/DALiuGE input parameters to those arguments.
+
+        This performs the following steps:
+
+        :param positionalArgs: postitional arguments of the function defined in self.func
+        :type positionalArgs: dict
+        :param keywordArguments: keyword arguments of the function defined in self.func
+        :type keywordArguments: dict
+
+        :return: funcargs, pargs; These are keyword and positional arguments to self.func
         """
         pargs = []  # positional arguments
         funcargs = {}  # Function arguments
-        vparg = []
-        vkarg = {}
 
         # update the positional args
-        pargsDict = self._initialise_positional_args(pargsDict)
-        if self._applicationArgs:
-            # if defined in both we use AppArgs values
-            for arg in self._applicationArgs:
-                # check value type and interpret
-                value, encoding = self._get_arg_info(arg)
-                if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
-                    try:
-                        value = ast.literal_eval(value)
-                        logger.debug(f"Evaluated %s to %s", value, type(value))
-                        self._applicationArgs[arg]["value"] = value
-                    except ValueError:
-                        logger.error(
-                            "Unable to evaluate %s", self._applicationArgs[arg]["value"]
-                        )
-                if arg in pargsDict:
-                    pargsDict[arg] = {"value": value, "encoding": encoding}
-                if arg in keyargsDict:
-                    keyargsDict[arg] = {"value": value, "encoding": encoding}
-
-            _ = [
-                self._applicationArgs.pop(k)
-                for k in pargsDict
-                if k in self._applicationArgs
-            ]
-            logger.debug("Updated posargs dictionary: %s", pargsDict)
-            logger.debug("Updated keyargs dictionary: %s", keyargsDict)
-
-            # Put all remaining arguments into *args and **kwargs
-            # TODO: This should only be done if the function signature allows it
-            logger.debug(f"Remaining AppArguments {self._applicationArgs}")
-            for arg in self._applicationArgs:
-                if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
-                    value = ast.literal_eval(self._applicationArgs[arg]["value"])
-                else:
-                    value = self._applicationArgs[arg]["value"]
-                if self._applicationArgs[arg]["positional"]:
-                    vparg.append(value)
-                else:
-                    vkarg[arg] = value
-        else:
-            logger.debug("AppArgs/pargsDict: %s", pargsDict)
-            encoding = self.input_parser if hasattr(self, "input_parser") else None
+        positionalArgsMap = self._initialise_args(positionalArgs)
+        keywordArgsMap = self._initialise_args(keywordArguments)
+        positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg =  (
+            self._populate_arguments_with_graph_data(positionalArgsMap, keywordArgsMap))
 
-        # TODO: check where this is defined in signature
-        self.arguments = inspect.getfullargspec(self.func)
-        if self.arguments.varargs:
+
+        # Determine if we can use the *args and **kwargs we identified above
+        if self.varargs or self.populate_arguments_without_signature:
             logger.debug("Adding remaining *args to pargs %s", vparg)
             pargs.extend(vparg)
-        if self.arguments.varkw:
+        if self.varkw:
             logger.debug("Adding remaining **kwargs to funcargs: %s", vkarg)
             funcargs.update(vkarg)
 
-        # Extract arg and values from pargs; we no longer need the encoding
-        logger.debug(f"Updating funcargs with values from pargsDict: {pargsDict}")
-        tmpPargs = {arg: subdict["value"] for arg, subdict in pargsDict.items()}
+        # Update any InputOutput ports that might have path names
+        for arg in input_outputs:
+            keywordArgsMap, positionalArgsMap = self._update_filepaths(
+                positionalArgsMap, keywordArgsMap, arg)
+
+        # Extract arg and values from pargs; we no longer need the metadata
+        logger.debug(f"Updating funcargs with values from pargsDict: {positionalArgsMap}")
+
+        tmpPargs = {argstr: argument.value for argstr, argument in
+                    positionalArgsMap.items()}
+
         funcargs.update(tmpPargs)
 
         # Mixin the values from named ports
-        portargs = self._ports2args(posargs, pargsDict, keyargsDict)
+        portargs = self._ports2args(positionalArgsMap, keywordArgsMap)
+
+        # Update any InputOutput ports that might have path names defined from input port
+        for arg in input_outputs:
+            keywordArgsMap, positionalArgsMap = self._update_filepaths(
+                positionalArgsMap, keywordArgsMap, arg)
 
         logger.debug(f"Updating funcargs with values from named ports {portargs}")
-        tmpPortArgs = {port: subdict["value"] for port, subdict in portargs.items()}
+        tmpPortArgs = {port: arg.value for port, arg in portargs.items()}
         funcargs.update(tmpPortArgs)
 
-        return [funcargs, pargs]
+        return funcargs, pargs
+
+    def _update_filepaths(
+            self,
+            positionalArgsMap: dict[str, Argument],
+            keywordArgsMap: dict[str, Argument],
+            arg: str
+    ):
+        """
+        Map any attribute that is an InputOutput
+
+        This is used to allow PyFunc apps that require a filename as input to refer to
+        the output data that is linked to the AppDrop. This resolves issues with us
+        hidden having side effects of functions saving files without DALiuGE being
+        aware of them.
+
+        Parameters
+        ----------
+        positionalArgsMap: map of positional argument strs (from function signature) to the
+        Argument objects (which store the current state of the values to be passed to
+        the function).
+        keywordArgsMap: map of keyword argument strs (from function signature) to the
+        Argument objects (which store the current state of the values to be passed to
+        the function).
+
+        Returns
+        -------
+        modified keywordArgsMap and positionalArgsMap
+        """
+
+        # match to output in the event that it is None
+        # Get the outputs accordingly
 
-    def _ports2args(self, posargs, pargsDict, keyargsDict) -> dict:
+        attr_uid_map = {}
+        output_port_count = {}
+        for output in self.parameters["outputs"]:
+            for key, value in output.items():
+                attr_uid_map[value] = key
+                if key in output_port_count:
+                    output_port_count[key] += 1
+                else:
+                    output_port_count[key] = 1
+
+        for arg_map in [keywordArgsMap, positionalArgsMap]:
+            if arg in arg_map:
+                output_uid = attr_uid_map[arg]
+                output_drop = next(
+                    (drop for drop in self.outputs if drop.uid == output_uid),
+                    None
+                )
+                argument = arg_map[arg]
+                parser = (DropParser(argument.encoding))
+                if parser == DropParser.PATH:
+                    argument.value = filepath_from_string(
+                        argument.value, dirname=output_drop.dirname, uid=output_drop.uid,
+                        humanKey=output_drop._humanKey
+                    )
+                    self._output_filepaths[output_uid] = argument.value
+                arg_map[arg] = argument
+                self.parameters[arg] = arg_map[arg].value
+        return keywordArgsMap, positionalArgsMap
+
+    def _ports2args(self, pargsDict, keyargsDict) -> dict:
         """
         Replace arguments with values from ports.
 
@@ -520,49 +656,44 @@ def _ports2args(self, posargs, pargsDict, keyargsDict) -> dict:
         """
         portargs = {}
         # 3. replace default argument values with named input ports
-        iitems = self._inputs
-        logger.debug("Mapping from _inputs: %s", iitems)
+        logger.debug("Mapping from _inputs: %s", self._inputs)
         logger.debug("Parameters: %s", self.parameters)
         if "inputs" in self.parameters and check_ports_dict(self.parameters["inputs"]):
             logger.debug("Mapping ports to inputs...")
-            check_len = min(
-                len(iitems),
-                self.fn_nargs + self.fn_nkw,
-            )
+            if self.fn_nargs == 0:
+                check_len = len(self._inputs)
+            else:
+                check_len = min(
+                    len(self._inputs),
+                    self.fn_nargs + self.fn_nkw,
+                )
             inputs_dict = collections.OrderedDict()
             for inport in self.parameters["inputs"]:
                 key = list(inport.keys())[0]
-                inputs_dict[key] = {
-                    "name": inport[key],
-                    "path": None,
-                    "drop": iitems[key],
-                }
-            parser = (
-                get_port_reader_function(self.input_parser)
-                if hasattr(self, "input_parser")
-                else None
-            )
-            portargs.update(
-                identify_named_ports(
-                    inputs_dict,
-                    posargs,
-                    pargsDict,
-                    keyargsDict,
-                    check_len=check_len,
-                    mode="inputs",
-                    addPositionalToKeyword=True,
-                    parser=parser,
-                )
+                inputs_dict[key] = {"name": inport[key], "path": None, "drop":
+                    self._inputs[key]}
+            keyPortArgs, posPortArgs = identify_named_ports(
+                inputs_dict,
+                pargsDict,
+                keyargsDict,
+                check_len=check_len,
+                mode="inputs",
+                addPositionalToKeyword=True,
+                parser=get_port_reader_function(self.input_parser)
             )
+            portargs.update(keyPortArgs)
         else:
-            for i, input_drop in enumerate(iitems.values()):
+            for i, input_drop in enumerate(self._inputs.values()):
                 parser = (
                     get_port_reader_function(self.input_parser)
                     if hasattr(self, "input_parser")
                     else None
                 )
                 value = parser(input_drop)
-                logger.debug("Port value pair: %s, %s", self.argnames[i], value)
+                if self.argnames:
+                    logger.debug("Port value pair: %s, %s",
+                                 self.argnames[i],
+                                 value)
 
         logger.debug(
             "Finally port mapping: %s, %s, %s", portargs, pargsDict, keyargsDict
@@ -577,21 +708,14 @@ def initialize_with_func_code(self):
         logger.debug(f"Initializing with func_code of type {type(self.func_code)}")
         if not isinstance(self.func_code, bytes):
             try:
-                self.func = import_using_code(
-                    self.func_code, self.func_name, serialized=False
-                )
-            except (SyntaxError, NameError) as err:
-                logger.warning(
-                    "Problem importing code: %s.",
-                    err,
-                )
-                # serialized = True
-        # if isinstance(self.func_code, bytes) or serialized:
-        #     if isinstance(self.func_code, str):
-        #         self.func_code = base64.b64decode(self.func_code.encode("utf8"))
-        #     self.func = import_using_code(
-        #         self.func_code, self.func_name, serialized=True
-        #     )
+                self.func = import_using_code(self.func_code, self.func_name,
+                    serialized=False)
+            except (SyntaxError, NameError):
+                serialized = True
+        if isinstance(self.func_code, bytes) or serialized:
+            if isinstance(self.func_code, str):
+                self.func_code = base64.b64decode(self.func_code.encode("utf8"))
+            self.func = import_using_code(self.func_code, self.func_name, serialized=True)
 
         self._init_fn_defaults()
         # make sure defaults are dicts
@@ -633,12 +757,13 @@ def initialize(self, **kwargs):
         self._clean_applicationArgs()
 
         self.num_args = len(
+            
             self._applicationArgs
         )  # number of additional arguments provided
 
         if not self.func_name and not self.func_code:
             raise InvalidDropException(
-                self, "No function specified (either via name or code)"
+                                       self, "No function specified (either via name or code)"
             )
         self.name = f"{self.name}:{self.func_name}"  # PyFuncApp is parent
         logger.info("AppDROP name: %s", self.name)
@@ -659,6 +784,7 @@ def initialize(self, **kwargs):
 
         # Mapping between argument name and input drop uids
         logger.debug(f"Input mapping provided: {self.func_arg_mapping}")
+        self._output_filepaths = {}
         self._recompute_data = {}
 
     def run(self):
@@ -692,8 +818,6 @@ def run(self):
         # if defaults dict has not been specified at all we'll go ahead anyway
 
         # 1. Fill arguments with rest of inputs
-        # TODO improve naming conventions
-        posargs = list(self.posonly.keys()) + list(self.poskw.keys())
         # fill the pargsDict with positional and poskw arguments and defaults
         pargsDict = {k: v.default for k, v in self.posonly.items()}
         pargsDict.update({k: v.default for k, v in self.poskw.items()})
@@ -707,22 +831,21 @@ def run(self):
         for arg in keyargsDict:
             keyargsDict[arg] = self.fn_defaults[arg]
 
-        # deal with arguments of any sort
-        funcargs, pargs = self._init_appArgs(pargsDict, keyargsDict, posargs)
+        # 2. Map arguments from user-defined data, or from InputPorts
+        funcargs, pargs = self._map_parameters_to_func_args(pargsDict, keyargsDict)
 
         self._recompute_data["args"] = funcargs.copy()
 
-        # 5. remove self argument if this is the initializer.
-        if (
-            self.func_name is not None
-            and self.func_name.split(".")[-1] in ["__init__", "__class__"]
-            and "self" in funcargs
-        ):
+        # 3. remove self argument if this is the initializer.
+        if (self.func_name is not None
+                and self.func_name.split(".")[-1] in ["__init__","__class__"]
+                and "self" in funcargs):
             funcargs.pop("self")
+
         logger.info(f"Running {self.func_name}")
         logger.debug(f"Arguments: *{pargs} **{funcargs}")
 
-        # 6. prepare for execution
+        # 4. prepare for execution
         # we capture and log whatever is produced on STDOUT
         capture = StringIO()
         try:
@@ -730,11 +853,15 @@ def run(self):
             logger.debug("Bound arguments: %s", bind)
         except TypeError as e:
             logger.error("Binding of arguments failed: %s", e)
-            raise
+        except AttributeError as e:
+            logger.debug("Binding failed due to signature not being callable")
 
-        # Here is where the function is actually executed
+        # 5. Here is where the function is actually executed
         with redirect_stdout(capture):
-            result = self.func(*bind.args, **bind.kwargs)
+            if isinstance(self.argsig, inspect.Signature):
+                result = self.func(*bind.args, **bind.kwargs)
+            else:
+                result = self.func(*pargs, **funcargs)
 
         logger.debug("Returned result from %s: %s", self.func_name, result)
         logger.info(
@@ -742,6 +869,7 @@ def run(self):
         )
         logger.debug(f"Finished execution of {self.func_name}.")
 
+        # 6. Process results
         # Depending on how many outputs we have we treat our result
         # as an iterable or as a single object. Each result is pickled
         # and written to its corresponding output
@@ -769,15 +897,24 @@ def _match_parser(self, output_drop):
     def write_results(self, result):
         from dlg.droputils import listify
 
-        if not self.outputs:
-            return
         result_iter = listify(result)
+        if not self.outputs and result_iter:
+            return
+
         logger.debug(
-            "Writing follow result to %d output: %s", len(self.outputs), result_iter
+            "Writing follow result to %d output: %s", len(self.outputs),
+                     result_iter
         )
         for i, o in enumerate(self.outputs):
-            # result = result_iter[0]
-            if len(result_iter) == 1:  # and len(self.outputs) > 1:
+            # Ensure that we don't produce two files for the same output DROP
+            if o.uid in self._output_filepaths:
+                # Trigger FileDROP filename update, but don't write to the drop because
+                # it has already been written to.
+                _ = o.getIO()
+                continue
+            if not result_iter and self.outputs:
+                result = result_iter
+            elif len(result_iter) == 1:
                 # We only have one element, no need to save as a list
                 result = result_iter[0]
             elif len(result_iter) > 1 and len(self.outputs) == 1:
@@ -790,12 +927,17 @@ def write_results(self, result):
                 result = result_iter[i]
 
             parser = self._match_parser(o)
-            if parser in [DropParser.PICKLE, DropParser.DILL]:
+            if parser is DropParser.PICKLE:
+                o.write(pickle.dumps(result))
+            elif parser is DropParser.DILL:
                 logger.debug(f"Writing dilled result {type(result)} to {o}")
                 o.write(dill.dumps(result))
             elif parser is DropParser.EVAL or parser is DropParser.UTF8:
-                encoded_result = repr(result).encode("utf-8")
-                o.write(encoded_result)
+                if isinstance(result, str):
+                    o.write(result.encode("utf-8"))
+                else:
+                    encoded_result = repr(result).encode("utf-8")
+                    o.write(encoded_result)
             elif parser is DropParser.NPY:
                 import numpy as np
 
@@ -820,3 +962,5 @@ def generate_recompute_data(self):
                 logger.debug(e)
                 self._recompute_data[name] = repr(val)
         return self._recompute_data
+
+
diff --git a/daliuge-engine/dlg/apps/simple_functions.py b/daliuge-engine/dlg/apps/simple_functions.py
index 3d30b7446..23e985a64 100644
--- a/daliuge-engine/dlg/apps/simple_functions.py
+++ b/daliuge-engine/dlg/apps/simple_functions.py
@@ -89,7 +89,11 @@ def string2json(string: str, pickle_flag: bool = False) -> list:
     """
     if not string:
         string = '""'
-    if not pickle_flag:
-        return json.loads(string)
-    else:
-        return pickle.dumps(json.loads(string))
+
+    try:
+        if not pickle_flag:
+            return json.loads(string)
+        else:
+            return pickle.dumps(json.loads(string))
+    except json.decoder.JSONDecodeError as e:
+        return [] if not pickle_flag else pickle.dumps([])
diff --git a/daliuge-engine/dlg/data/drops/data_base.py b/daliuge-engine/dlg/data/drops/data_base.py
index 624dd37e8..ba2b1fa62 100644
--- a/daliuge-engine/dlg/data/drops/data_base.py
+++ b/daliuge-engine/dlg/data/drops/data_base.py
@@ -97,7 +97,7 @@ def decrRefCount(self):
         with self._refLock:
             self._refCount -= 1
 
-    # @track_current_drop
+    @track_current_drop
     def open(self, **kwargs):
         """
         Opens the DROP for reading, and returns a "DROP descriptor"
@@ -366,6 +366,8 @@ def _map_input_ports_to_params(self):
         finalDropPortMap = {}  # Final mapping of named port to value stored in producer
 
         for p in self.producers:
+            producerUid = p.uid
+            producerPortValueMap[producerUid] = {}
             params = p.parameters["outputs"]
             for param in params:
                 try:
@@ -375,15 +377,9 @@ def _map_input_ports_to_params(self):
                     continue
                 portName = param[key]
                 portValue = ""
-                producerUid = p.uid
                 if portName in p.parameters:
                     portValue = p.parameters[param[key]]
-                # TODO This currently only allows 1 UID -> Portname/Value
-                # Investigate UID -> [Portname1:Value1, Portnam2:value2,..,]
-                producerPortValueMap[producerUid] = {
-                    "portname": portName,
-                    "value": portValue,
-                }
+                producerPortValueMap[producerUid][portName] = portValue
 
         for port in dropInputPorts:
             try:
@@ -391,21 +387,19 @@ def _map_input_ports_to_params(self):
             except AttributeError:
                 logging.debug("Producer %s does not have named ports", p.uid)
                 continue
-            for uid, portname in port.items():
+            for uid, input_port_name in port.items():
                 try:
-                    logger.debug("Trying to map UID: %s to port %s", uid, portname)
-                    tmp = producerPortValueMap[uid]
-                    if tmp["portname"] == portname:
-                        finalDropPortMap[portname] = tmp["value"]
+                    ouput_port_name = self.parameters["port_map"][input_port_name]
+                    if ouput_port_name in producerPortValueMap[uid]:
+                        finalDropPortMap[input_port_name] = producerPortValueMap[uid][
+                            ouput_port_name]
                 except KeyError:
-                    logger.debug("UID %s Not available!", uid)
+                    logging.warning("%s not available.", input_port_name)
 
         for portname in finalDropPortMap:
             if portname in self.parameters:
                 self.parameters[portname] = finalDropPortMap[portname]
 
-        self._updatedPorts = True
-
     @abstractmethod
     def getIO(self) -> DataIO:
         """
diff --git a/daliuge-engine/dlg/data/drops/file.py b/daliuge-engine/dlg/data/drops/file.py
index 51e55d964..f9fb44f3a 100644
--- a/daliuge-engine/dlg/data/drops/file.py
+++ b/daliuge-engine/dlg/data/drops/file.py
@@ -22,13 +22,13 @@
 import errno
 import logging
 import os
-import re
 
 from dlg.common.reproducibility.reproducibility import common_hash
+from dlg.data import path_builder
+from dlg.data.io import FileIO
 from dlg.ddap_protocol import DROPStates
 from .data_base import DataDROP, PathBasedDrop, logger, track_current_drop
 from dlg.exceptions import InvalidDropException
-from dlg.data.io import FileIO
 from dlg.meta import dlg_bool_param
 from dlg.utils import isabs
 from typing import Union
@@ -106,7 +106,6 @@ def sanitize_paths(self, filepath: str) -> Union[None, str]:
             filepath = self.get_dir(filepath)
         return filepath
 
-    non_fname_chars = re.compile(r":|%s" % os.sep)
 
     def initialize(self, **kwargs):
         """
@@ -117,10 +116,12 @@ def initialize(self, **kwargs):
 
     def _setupFilePaths(self):
         filepath = self.parameters.get("filepath", None)
+        # TODO ADD SUFFIX/PREFIX
         dirname = None
         filename = None
 
         if filepath:  # if there is anything provided
+            # TODO do f-string substitution if necessary
             if "/" not in filepath:  # just a name
                 filename = filepath
                 dirname = self.get_dir(".")
@@ -145,12 +146,11 @@ def _setupFilePaths(self):
 
         # Default filename to drop human readable format based on UID
         if filename is None:
-            # '2024-10-30T12:01:57_0140555b-8c23-4d6a-9e24-e16c15555e8c_0'
-            fn = self.uid.split("_")[0] + "_" + str(self._humanKey)
-            filename = self.non_fname_chars.sub("_", fn)
+            filename = path_builder.base_uid_filename(self.uid, self._humanKey)
+
+
         self.filename = filename
         self.dirname = self.get_dir(dirname)
-
         self._root = self.dirname
         self._path = (
             os.path.join(self.dirname, self.filename) if self.filename else self.dirname
@@ -168,7 +168,8 @@ def _setupFilePaths(self):
     def getIO(self):
 
         # We need to update named_ports now we have runtime information
-        if not self._updatedPorts:
+        # if not self._updatedPorts:
+        if not self.parameters.get("filepath", None):
             self._map_input_ports_to_params()
             self._setupFilePaths()
 
diff --git a/daliuge-engine/dlg/data/path_builder.py b/daliuge-engine/dlg/data/path_builder.py
new file mode 100644
index 000000000..4facb9e5b
--- /dev/null
+++ b/daliuge-engine/dlg/data/path_builder.py
@@ -0,0 +1,147 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+"""
+Module with utility methods to create runtime file paths for the DALiuGE Engine.
+
+The Engine supports implicit filepath naming if FileDROPs have no user-defined name in
+the Logical Graph Template. This helps when multiple FileDROPs are created in a
+construct (e.g. Scatter or Loop) and we don't run the risk of overwriting them.
+
+This module provides the methods that support different filepath naming heuristics to
+facilitate implicit filepath naming at runtime.
+"""
+import datetime
+import logging
+import re
+import os
+import uuid
+
+NON_FILENAME_CHARACTERS = re.compile(fr":|{os.sep}")
+
+def default_map():
+    """
+    Get the default map for the FSTRING replacement keywords
+    """
+    return {
+        "dlg": "DALIGUE",
+        "datetime": datetime.date.today().strftime("%Y-%m-%d"),
+        "uid": str(uuid.uuid4()),
+    } 
+
+def base_uid_filename(uid: str, humanKey: str):
+    """
+    This a basic filename generator, using the UID and humandReadableKey. The function
+    returns only the name of the file, and expects the full filepath to be handled by
+    the function caller.
+
+    Parameters
+    ----------
+    uid: str, the UID of the DROP
+    humanKey: str, a human-readable key (typically a truncated UID) for user-facing
+        display
+
+    Notes
+    -----
+    An example DROP UID would be:
+
+        '2024-10-30T12:01:57_0140555b-8c23-4d6a-9e24-e16c15555e8c_0'
+
+    This approach will lead to an abridged UID along with the humanKey appended:
+
+       '2024-10-30T12:01:57_0140555b_0/1/1'
+
+    In this case, 0/1/1 is the "humanKey", derived from the DALiuGE Translator
+    constructs (For more information on humanKey generation, refer to
+    dlg.common.truncateUidToKey).
+
+    The final part of the method replaces characters that cannot be in filenames from
+    the UIDs (e.g. "/").
+
+    Returns
+    -------
+        str: Base filename
+    """
+    if not uid:
+        return None
+    if not humanKey:
+        humanKey = ""
+    fn = str(uid).split("_")[0] + "_" + str(humanKey)
+    return NON_FILENAME_CHARACTERS.sub("_", fn)
+
+def find_dlg_fstrings(filename: str) -> list[str]:
+    """
+    Find f-string style braces intended for replacement by the Engine and return a list
+    of the placeholder strings.
+
+    Supports escaping the braces by using two in a row (e.g. {{example}}).
+
+    :param filename: filename with possible f-string brackets for keyword replacement
+
+    .. note::
+        The following cases are supported by this method:
+        1. No braces:
+            "unique_file.txt"  # []
+
+        2. Standard {} replacement :
+            "gather_{uid}.py" -> return ['uid']
+
+        3. Escaped braces:
+            "gather_{{uid}}.py"  -> []
+
+        4. Multiple braces
+            "prefix_{uid}_{datetime}.dat"  # ['uid', 'datetime']
+
+    :return: list of placeholder strings (incl. {})
+    """
+    opts = []
+    try:
+        opts.extend(re.findall(r'\{([^{}]+)\}', filename))
+        return opts
+    except TypeError as e:
+        logging.warning("Data not in expected format %s",e)
+        return opts
+
+
+def filepath_from_string(filename: str, dirname: str = "", **kwargs) -> str:
+    """
+    Attempts to construct a filename from filename and possible mappings, which are
+    built from a combination of FSTRING_MAP and **kwargs.
+
+    Returns
+    -------
+    filename
+    """
+
+    opts = []
+    fstring_map = default_map() 
+    fstring_map.update(kwargs)
+    if not filename and "humanKey" in fstring_map:
+        return base_uid_filename(fstring_map["uid"], fstring_map["humanKey"])
+    elif not filename:
+        return filename
+
+    opts.extend(find_dlg_fstrings(filename))
+    for fp in opts:
+        filename = filename.replace(f"{{{fp}}}", fstring_map[fp])
+
+    return f"{dirname}/{filename}" if dirname else filename
diff --git a/daliuge-engine/dlg/drop_loaders.py b/daliuge-engine/dlg/drop_loaders.py
index e5ffb05af..15119227c 100644
--- a/daliuge-engine/dlg/drop_loaders.py
+++ b/daliuge-engine/dlg/drop_loaders.py
@@ -111,6 +111,8 @@ def load_dill(drop: "DataDROP"):
     """
     buf = io.BytesIO()
     desc = drop.open()
+    if drop.size == 0:
+        return None
     while True:
         data = drop.read(desc)
         if not data:
diff --git a/daliuge-engine/dlg/named_port_utils.py b/daliuge-engine/dlg/named_port_utils.py
index 96ad4dc9e..76898c84e 100644
--- a/daliuge-engine/dlg/named_port_utils.py
+++ b/daliuge-engine/dlg/named_port_utils.py
@@ -1,14 +1,30 @@
 import ast
-from enum import Enum
 import logging
 import collections
 from dlg.data.drops.data_base import DataDROP
 import dlg.droputils as droputils
+from dlg.data import path_builder
 import dlg.drop_loaders as drop_loaders
-from typing import Tuple
+
+from dataclasses import dataclass
+from enum import Enum, IntEnum, auto
+from typing import Tuple, Union
+
+from dlg.drop import AbstractDROP
 
 logger = logging.getLogger(f"dlg.{__name__}")
 
+class ArgType(IntEnum):
+    """
+    Arguments can be positional or keyword.
+
+    This enum is used when determining what we do with the argument, as POSITIONAL
+    arguments will need to be treated differently as the order in which they appear
+    is important.
+    """
+    POSITIONAL = auto()
+    KEYWORD = auto()
+
 
 class DropParser(Enum):
     RAW = "raw"
@@ -23,6 +39,23 @@ class DropParser(Enum):
     UTF8 = "utf-8"
 
 
+@dataclass()
+class Argument:
+    """
+    Encapsulate data associated with all arguments.
+
+    There are a few things here that will always be the default,
+    but it makes more sense to have the same information always true across all instances
+    of an Argument, rather than create special cases for specific dictionary elements
+    """
+
+    value: object
+    encoding: DropParser = DropParser.DILL
+    type: Union [ArgType, None] = None
+    precious: bool = False
+    positional: bool = False
+
+
 def serialize_kwargs(keyargs, prefix="--", separator=" "):
     kwargs = {}
     for name, value in iter(keyargs.items()):
@@ -83,17 +116,15 @@ def serialize_applicationArgs(applicationArgs, prefix="--", separator=" "):
     logger.info("Constructed command line arguments: %s %s", pargs, kwargs)
     return (pargs, kwargs)
 
-
 def identify_named_ports(
     port_dict: dict,
     positionalArgs: list,
-    positionalPortArgs: dict,
     keywordArgs: dict,
     check_len: int = 0,
     mode: str = "inputs",
     parser: callable = None,
     addPositionalToKeyword: bool = False,
-) -> dict:
+) -> tuple[dict, dict]:
     """
     Checks port names for matches with arguments and returns mapped ports.
 
@@ -130,6 +161,7 @@ def identify_named_ports(
     )
     logger.debug("Checking against keyargs: %s", keywordArgs)
     keywordPortArgs = {}
+    positionalPortArgs = collections.OrderedDict(positionalArgs)
     positionalArgs = list(positionalArgs)
     keys = list(port_dict.keys())
     logger.debug("Checking ports: %s against %s %s", keys, positionalArgs, keywordArgs)
@@ -144,7 +176,7 @@ def identify_named_ports(
             value = ""  # make sure we are passing NULL drop events
         if key in positionalArgs:
             try:
-                encoding = DropParser(positionalPortArgs[key]["encoding"])
+                encoding = DropParser(positionalPortArgs[key].encoding)
             except ValueError:
                 logger.warning("No encoding set for %key: possible default")
                 continue
@@ -152,7 +184,7 @@ def identify_named_ports(
             if parser:
                 logger.debug("Reading from %s encoded port using %s", encoding, parser.__repr__())
                 value = parser(port_dict[keys[i]]["drop"])
-            positionalPortArgs[key]["value"] = value
+            positionalPortArgs[key].value = value
             logger.debug("Using %s '%s' for parg %s", mode, value, key)
             positionalArgs.remove(key)
             # We have positional argument that is also a keyword
@@ -160,7 +192,7 @@ def identify_named_ports(
                 keywordPortArgs.update({key: positionalPortArgs[key]})
         elif key in keywordArgs:
             try:
-                encoding = DropParser(keywordArgs[key]["encoding"])
+                encoding = DropParser(keywordArgs[key].encoding)
             except ValueError:
                 logger.warning("No encoding set for %key: possible default")
                 continue
@@ -170,7 +202,7 @@ def identify_named_ports(
                 value = parser(port_dict[keys[i]]["drop"])
             # if not found in appArgs we don't put them into portargs either
             # pargsDict.update({key: value})
-            keywordArgs[key]["value"] = value
+            keywordArgs[key].value = value
             keywordPortArgs.update({key: keywordArgs[key]})
             logger.debug("Using %s of type %s for kwarg %s", mode, type(value), key)
             _ = keywordArgs.pop(key)  # remove from original arg list
@@ -184,7 +216,7 @@ def identify_named_ports(
             )
 
     logger.debug("Returning kw mapped ports: %s", keywordPortArgs)
-    return keywordPortArgs
+    return keywordPortArgs, positionalPortArgs
 
 
 def check_ports_dict(ports: list) -> bool:
@@ -257,6 +289,9 @@ def replace_named_ports(
 
     positionalArgs = _get_args(appArgs, positional=True)
     keywordArgs = _get_args(appArgs, positional=False)
+    # we will need an ordered dict for all positional arguments
+    # thus we create it here and fill it with values
+
     logger.debug(
         "posargs: %s; keyargs: %s, %s",
         positionalArgs,
@@ -271,29 +306,29 @@ def replace_named_ports(
 
     # Update the argument dictionaries in-place based on the port names.
     # This needs to be done for both the input ports and output ports on the drop.
-    _process_port(
+    tmp_key, tmp_port = _process_port(
         inport_names,
         inputs_dict,
-        keywordPortArgs,
         positionalArgs,
-        positionalPortArgs,
         keywordArgs,
         iitems,
         parser,
         "inputs",
     )
+    keywordPortArgs.update(tmp_key)
+    positionalPortArgs.update(tmp_port)
 
-    _process_port(
+    tmp_key, tmp_port =_process_port(
         outport_names,
         outputs_dict,
-        keywordPortArgs,
         positionalArgs,
-        positionalPortArgs,
         keywordArgs,
         oitems,
         parser,
         "outputs",
     )
+    keywordPortArgs.update(tmp_key)
+    positionalPortArgs.update(tmp_port)
 
     logger.debug("Arguments from ports: %s, %s,", keywordPortArgs, positionalPortArgs)
 
@@ -303,11 +338,14 @@ def replace_named_ports(
     keywordArgs = _get_args(appArgs, positional=False)
 
     # Extract values from dictionaries - "encoding" etc. are irrelevant
-    keywordArgs = {arg: subdict['value'] for arg, subdict in keywordArgs.items()}
-    positionalArgs = {arg: subdict['value'] for arg, subdict in positionalArgs.items()}
-    keywordPortArgs = {arg: subdict['value'] for arg, subdict in keywordPortArgs.items()}
-    positionalPortArgs = {arg: subdict['value'] for arg, subdict in
-                          positionalPortArgs.items()}
+    positionalArgs = {argstr: argument.value for argstr, argument in (
+            positionalArgs.items())}
+    keywordArgs = {argstr: argument.value for argstr, argument in keywordArgs.items()}
+    keywordPortArgs = {
+        argstr: argument.value for argstr, argument in keywordPortArgs.items()
+    }
+    positionalPortArgs = {argstr: argument.value for argstr, argument in (
+        positionalPortArgs.items())}
 
     #  Construct the final keywordArguments and positionalPortArguments
     for k, v in keywordPortArgs.items():
@@ -327,9 +365,7 @@ def replace_named_ports(
 def _process_port(
     port_names,
     ports,
-    keywordPortArgs,
     positionalArgs,
-    positionalPortArgs,
     keywordArgs,
     iitems,
     parser,
@@ -345,33 +381,37 @@ def _process_port(
     identify_named_ports() method.
     """
 
+    keywordPortArgs = {}
+    positionalPortArgs = {}
     if check_ports_dict(port_names):
         for port in port_names:
             key = list(port.keys())[0]
             ports[key].update({"name": port[key]})
-        keywordPortArgs.update(
-            identify_named_ports(
-                ports,
-                positionalArgs,
-                positionalPortArgs,
-                keywordArgs,
-                check_len=len(iitems),
-                mode=mode,
-                parser=parser,
-            )
+        keywordPortArgs, positionalPortArgs = identify_named_ports(
+            ports,
+            positionalArgs,
+            keywordArgs,
+            check_len=len(iitems),
+            mode=mode,
+            parser=parser,
         )
+
     else:
         for i in range(min(len(iitems), len(positionalArgs))):
             keywordPortArgs.update({list(positionalArgs)[i]: list(iitems)[i][1]})
 
+    return keywordPortArgs, positionalPortArgs
 
 def _get_args(appArgs, positional=False):
     """
     Separate out the arguments dependening on if we want positional or keyword style
     """
     args = {
-        arg: {"value": appArgs[arg]["value"],
-              "encoding": appArgs[arg].get("encoding", "dill")}
+        arg: Argument(
+            value=appArgs[arg]["value"],
+            encoding= appArgs[arg].get("encoding", "dill"),
+            positional=positional
+        )
         for arg in appArgs
         if (appArgs[arg]["positional"] == positional)
     }
@@ -415,13 +455,21 @@ def utf8decode(drop: "DataDROP"):
     elif input_parser is DropParser.NPY:
         reader = drop_loaders.load_npy
     elif input_parser is DropParser.PATH:
-        reader = lambda x: x.path
+        def PathFromData(x: AbstractDROP):
+            # Attempt to access path from DROP "x"
+            # If not Path, this could be a memory Drop with path information.
+            # If so, try and decode accordingly.
+            try:
+                return x.path
+            except AttributeError:
+                return drop_loaders.load_utf8(x)
+        reader = PathFromData
     elif input_parser is DropParser.DATAURL:
         reader = lambda x: x.dataURL
     elif input_parser is DropParser.DILL:
         reader = drop_loaders.load_dill
     elif input_parser is DropParser.BINARY:
-        reader = drop_loaders.load_binary
+         reader = drop_loaders.load_binary
     else:
         raise ValueError("Invalid input parser specified: %s", input_parser.__repr__())
     return reader
diff --git a/daliuge-engine/setup.py b/daliuge-engine/setup.py
index a20228aef..f51b41092 100644
--- a/daliuge-engine/setup.py
+++ b/daliuge-engine/setup.py
@@ -179,7 +179,9 @@ def run(self):
     "MPI": ["mpi4py"],
     # AWS storage types
     "aws": ["boto3"],
-    "test": ["pytest", "eagle-test-graphs"],
+    "test": ["pytest",
+            "eagle-test-graphs @ git+https://github.com/ICRAR/EAGLE_test_repo@LIU-5"]
+             # "eagle-test-graphs==0.1.6"],
 }
 
 setup(
diff --git a/daliuge-engine/test/apps/test_bash.py b/daliuge-engine/test/apps/test_bash.py
index 5eac4ba14..111aa1e46 100644
--- a/daliuge-engine/test/apps/test_bash.py
+++ b/daliuge-engine/test/apps/test_bash.py
@@ -184,7 +184,7 @@ def test_single_pipe(self):
         c.addOutput(d)
 
         # Let's fire the app
-        with DROPWaiterCtx(self, d, 2):
+        with DROPWaiterCtx(self, d, 200):
             a.async_execute()
 
         # The application executed, finished, and its output was recorded
diff --git a/daliuge-engine/test/dlg_end_to_end_utils.py b/daliuge-engine/test/dlg_end_to_end_utils.py
new file mode 100644
index 000000000..ecbe5db70
--- /dev/null
+++ b/daliuge-engine/test/dlg_end_to_end_utils.py
@@ -0,0 +1,68 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2015, 2024
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+"""
+Utility classes and functions designed to facilitate end-to-end testing of DALiuGE.
+
+This requires both the daliuge-engine and daliuge-translator to be installed.
+"""
+import json
+import pytest
+import unittest
+
+import dlg.droputils as droputils
+import dlg.graph_loader as graph_loader
+
+from pathlib import Path
+from dlg.common.path_utils import get_lg_fpath
+# Note this test will only run with a full installation of DALiuGE
+pexpect = pytest.importorskip("dlg.dropmake")
+
+def create_and_run_graph_spec_from_graph_file(test_case: unittest.TestCase,
+                                              graph_name: str,
+                                              resolve_path = True,
+                                              app_root=True):
+    """
+    Boilerplate graph running code which takes a Logical Graph Template (LGT) and
+    performs the translation and submission through the graph loader.
+
+    Returns drop leaf nodes on graph completion.
+    """
+    if resolve_path:
+        spec = Path(get_lg_fpath("drop_spec", graph_name))
+    else:
+        spec = Path(graph_name)
+    with spec.open('r', encoding="utf-8") as f:
+            appDropSpec = json.load(f)
+
+    roots = graph_loader.createGraphFromDropSpecList(appDropSpec)
+    # drops = [v for d,v in drops.items()]
+    leafs = droputils.getLeafNodes(roots)
+    with droputils.DROPWaiterCtx(test_case, leafs, timeout=300):
+        for drop in roots:
+            if app_root:
+                fut = drop.async_execute()
+                fut.result()
+            else:
+                drop.setCompleted()
+
+    return leafs
diff --git a/daliuge-engine/test/dlg_engine_testutils.py b/daliuge-engine/test/dlg_engine_testutils.py
index 47ed90f76..fedf94fc0 100644
--- a/daliuge-engine/test/dlg_engine_testutils.py
+++ b/daliuge-engine/test/dlg_engine_testutils.py
@@ -25,15 +25,12 @@
 import http.client
 import json
 import random
-import unittest
 
 from time import sleep
 
 from dlg import utils
 from dlg import droputils
-from dlg.common import dropdict
-from dlg.ddap_protocol import DROPStates, DROPRel, DROPLinkType
-from dlg.apps.app_base import BarrierAppDROP
+from dlg.ddap_protocol import DROPStates
 from dlg.manager.node_manager import NodeManager
 from dlg.manager.manager_data import Node
 
@@ -217,3 +214,5 @@ def _test_runGraphInTwoNMs(
         dm1.destroySession(sessionId)
         dm2.destroySession(sessionId)
         return leaf_drop_data
+
+
diff --git a/daliuge-engine/test/named_ports/__init__.py b/daliuge-engine/test/named_ports/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/daliuge-engine/test/named_ports/test_filedrops.py b/daliuge-engine/test/named_ports/test_filedrops.py
new file mode 100644
index 000000000..345a2ad7d
--- /dev/null
+++ b/daliuge-engine/test/named_ports/test_filedrops.py
@@ -0,0 +1,97 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+
+"""
+This module is used to test FileDROP input/output naming through named ports.
+"""
+import datetime
+import os
+import shutil
+
+import dill
+import pytest
+import unittest
+
+from importlib.resources import files
+from pathlib import Path
+
+from dlg.ddap_protocol import DROPStates
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
+from daliuge_tests.engine import test_filedrops as test_graphs
+
+# Note this test will only run with a full installation of DALiuGE.
+pexpect = pytest.importorskip("dlg.dropmake")
+
+class TestBasicApp(unittest.TestCase):
+    """
+    Test that the Graph produces:
+
+        - A file as a side effect of the behaviour of the function ((hello_{dt}.txt)).
+        - A file that is a result of copying the side effect filename, using the FileDROP
+          connected to the AppDrop.
+        - A file that is a result of the output of the function (the return)
+    """
+
+    graph = f"{files(test_graphs)}/Filepath_test_side_effectPG.graph"
+    rundir = "/tmp/daliuge_tfiles"
+
+    def setUp(self):
+        if os.path.exists(self.rundir):
+            shutil.rmtree(self.rundir)
+        # if os.path.exists(self.rundir):
+        #     shutil.rmtree(self.rundir)
+        # os.makedirs(self.rundir)
+
+    # def tearDown(self):
+    #     if os.path.exists(self.rundir):
+    #         shutil.rmtree(self.rundir)
+
+    def test_pyfunc_to_fileapp(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False, app_root=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
+
+        dt = datetime.date.today().strftime("%Y-%m-%d")
+        leaf = leafs.pop()
+
+        hello_filename = f"hello_{dt}.txt"
+        hello_text = "Hello Ryan"
+        hello_duplicate = "hello2.txt"
+
+        output_filename = "output.txt"
+        output_text = "Hello from write_file"
+
+        expected_files = [hello_filename, hello_duplicate, output_filename]
+        created_files = [p.name for p in Path(leaf.dirname).iterdir()]
+        for e in expected_files:
+            self.assertTrue(e in created_files)
+
+        for p in Path(leaf.dirname).iterdir():
+            if p.name == hello_filename or p.name == hello_duplicate:
+                text = hello_text
+            else:
+                text = output_text
+                with p.open() as fp:
+                    self.assertEqual(text, fp.read())
diff --git a/daliuge-engine/test/test_named_port_apps.py b/daliuge-engine/test/named_ports/test_named_port_apps.py
similarity index 65%
rename from daliuge-engine/test/test_named_port_apps.py
rename to daliuge-engine/test/named_ports/test_named_port_apps.py
index f6d93e20e..18c83206e 100644
--- a/daliuge-engine/test/test_named_port_apps.py
+++ b/daliuge-engine/test/named_ports/test_named_port_apps.py
@@ -34,7 +34,8 @@
 import logging
 import unittest
 import pytest
-import json
+
+import daliuge_tests.engine.simple_functions as simple_functions
 
 logging.basicConfig(level=logging.DEBUG)
 logger = logging.getLogger("__main__." + __name__)
@@ -42,44 +43,10 @@
 # Note this test will only run with a full installation of DALiuGE.
 pexpect = pytest.importorskip("dlg.dropmake")
 
-from pathlib import Path
 from importlib.resources import files
 
-import dlg.droputils as droputils
-import dlg.graph_loader as graph_loader
-
 from dlg.ddap_protocol import DROPStates
-from dlg.dropmake import path_utils
-
-
-def create_and_run_graph_spec_from_lgt(
-    test_case: unittest.TestCase, graph_name: str, resolve_path=True, app_root=True
-):
-    """
-    Boilerplate graph running code which takes a Logical Graph Template (LGT) and
-    performs the translation and submission through the graph loader.
-
-    Returns drop leaf nodes on graph completion.
-    """
-    if resolve_path:
-        spec = Path(path_utils.get_lg_fpath("drop_spec", graph_name))
-    else:
-        spec = Path(graph_name)
-    with spec.open("r", encoding="utf-8") as f:
-        appDropSpec = json.load(f)
-
-    roots = graph_loader.createGraphFromDropSpecList(appDropSpec)
-    # drops = [v for d,v in drops.items()]
-    leafs = droputils.getLeafNodes(roots)
-    with droputils.DROPWaiterCtx(test_case, leafs, timeout=3):
-        for drop in roots:
-            if app_root:
-                fut = drop.async_execute()
-                fut.result()
-            else:
-                drop.setCompleted()
-
-    return leafs
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
 
 
 class TestPortsEncoding(unittest.TestCase):
@@ -111,7 +78,7 @@ def test_pyfunc_ports_encoding(self):
                     ------> pickle(8) ---             -----> pickle(8)
         """
 
-        leafs = create_and_run_graph_spec_from_lgt(self, "test_ports.graph")
+        leafs = create_and_run_graph_spec_from_graph_file(self, "test_ports.graph")
         for l in leafs:
             self.assertEqual(DROPStates.COMPLETED, l.status)
 
@@ -123,46 +90,51 @@ def test_pyfunc_ports_encoding(self):
         desc = leaf.open()
         self.assertEqual("array(2)", leaf.read(desc).decode())
 
-    @unittest.skip
     def test_bash_shell_ports(self):
         """
         "drop_spec", "pyfunc_glob_shell_test.graph"
         """
-        leafs = create_and_run_graph_spec_from_lgt(self, "pyfunc_glob_shell_test.graph")
-
+        leafs = create_and_run_graph_spec_from_graph_file(self, "pyfunc_glob_shell_test.graph")
+        
         for l in leafs:
             self.assertEqual(DROPStates.COMPLETED, l.status)
         self.assertEqual(0, leafs.pop().proc.returncode)
 
+class TestString2JsonGraphs(unittest.TestCase):
+    """
+    Test variations in edge cases for string2json functions
 
-class TestSimpleFunctionGraphs(unittest.TestCase):
+    Func: dlg.apps.simple_functions.string2json
+    """
 
-    @unittest.skip
     def test_string2json(self):
         """
-        Test variations in edge cases for string2json functions:
-            - No parameters are provided in graph (string2jsonPGT.graph)
-            - Empty parameters are provided in graph
-            - Incorrect parameters are provided in graph (e.g. component instead of
-               application)
-
-        Func: dlg.apps.simple_functions.string2json
-
+        No parameters are provided in graph (string2jsonPG.graph)
         """
-
-        import daliuge_tests.engine.simple_functions as simple_functions
-
         graph = f"{files(simple_functions)}/string2jsonPG.graph"
-        leafs = create_and_run_graph_spec_from_lgt(self, graph, resolve_path=False)
+        leafs = create_and_run_graph_spec_from_graph_file(self, graph, resolve_path=False)
         for l in leafs:
             self.assertEqual(DROPStates.COMPLETED, l.status)
+
+    def test_string2json_incorrectArgs(self):
+        """
+        Empty parameters are provided in graph
+        """
         graph = f"{files(simple_functions)}/string2json_incorrect_argsPG.graph"
-        leafs = create_and_run_graph_spec_from_lgt(self, graph, resolve_path=False)
+        leafs = create_and_run_graph_spec_from_graph_file(self, graph, resolve_path=False)
         for l in leafs:
             self.assertEqual(DROPStates.COMPLETED, l.status)
+
+    def test_string2json_incorrect_parameterType(self):
+        """
+        Incorrect parameters are provided in graph (e.g. component instead of application)
+        :return:
+        """
         graph = f"{files(simple_functions)}/string2json_incorrect_paramtypePGT.graph"
-        leafs = create_and_run_graph_spec_from_lgt(
-            self, graph, resolve_path=False, app_root=False
-        )
+        leafs = create_and_run_graph_spec_from_graph_file(self,
+                                                          graph,
+                                                          resolve_path=False,
+                                                          app_root=False)
         for l in leafs:
             self.assertEqual(DROPStates.COMPLETED, l.status)
+
diff --git a/daliuge-engine/test/named_ports/test_path_builder.py b/daliuge-engine/test/named_ports/test_path_builder.py
new file mode 100644
index 000000000..f9017e6e5
--- /dev/null
+++ b/daliuge-engine/test/named_ports/test_path_builder.py
@@ -0,0 +1,78 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+import unittest
+import datetime
+from dlg.data.path_builder import (find_dlg_fstrings, filepath_from_string,
+                                    base_uid_filename)
+
+
+class TestPathBuilders(unittest.TestCase):
+
+    def setUp(self):
+
+        self.filename = "test"
+
+
+    def testBaseUIDGenerator(self):
+        """
+        Using the baseUID Generator with the humanreadable key
+        """
+        res = base_uid_filename("", "")
+        self.assertEqual(None, res)
+        res = base_uid_filename(123456, None)
+        self.assertEqual("123456_", res)
+        res = base_uid_filename("123456_abcdef", "0/1")
+        self.assertEqual("123456_0_1", res)
+
+
+
+    def test_file_path_from_string(self):
+        uid = "123456"
+        res = filepath_from_string("prefix_{uid}_{datetime}.dat", uid=uid)
+        dstr = datetime.date.today().strftime("%Y-%m-%d")
+        self.assertEqual(f"prefix_123456_{dstr}.dat", res)
+        dt = "2025-08-17"
+        res = filepath_from_string("prefix_{uid}_{datetime}.dat",
+                                   uid=uid, datetime=dt)
+        self.assertEqual("prefix_123456_2025-08-17.dat", res)
+        res = filepath_from_string(None, uid=None)
+        self.assertEqual(None, res)
+
+
+class TestHelperFunctions(unittest.TestCase):
+
+    def test_find_dlg_strings_normal(self):
+        fn_empty = ""
+        self.assertEqual([], find_dlg_fstrings(fn_empty))
+        fn = "prefix_{uid}_{datetime}.dat"  # ['uid', 'datetime']
+        self.assertEqual(['uid', 'datetime'], find_dlg_fstrings(fn))
+        fn = "prefix_{{uid}}.txt"
+        self.assertEqual(["uid"], find_dlg_fstrings(fn))
+
+    def test_find_dlg_strings_badinput(self):
+        fn = 57
+        self.assertRaises(TypeError, find_dlg_fstrings(fn))
+
+        fn = None
+        self.assertRaises(TypeError, find_dlg_fstrings(fn))
+
diff --git a/daliuge-translator/dlg/dropmake/lg.py b/daliuge-translator/dlg/dropmake/lg.py
index 4248efc2f..2cdefbb4f 100644
--- a/daliuge-translator/dlg/dropmake/lg.py
+++ b/daliuge-translator/dlg/dropmake/lg.py
@@ -494,33 +494,22 @@ def _link_drops(
             self._drop_dict["new_added"].append(dropSpec_null)
         elif s_type in ["Application", "Control"]:
             sname = slgn._getPortName("outputPorts", index=-1)
-            tname = tlgn._getPortName("inputPorts")
+            tname = tlgn._getPortName("inputPorts", index=-1)
 
             # sname is dictionary of all output ports on the sDROP.
             # Therefore there's an expected number of output edges that we need to add
-            # We keep track of this by querying the sDROP's current outputs and seeing
-            # what is currently 'lowest' in number. Eventual, all output port names will
+            # We keep track of this by querying the sDROP's current outputs and seeing 
+            # what is currently 'lowest' in number. Eventual, all output port names will 
             # be added.
-            expected = [p for p in sname.keys()]
-            output_port = expected[0]
-            if "outputs" in sdrop:
-                actual = []
-                [actual.extend(pair.values()) for pair in sdrop["outputs"]]
-                actual_counts = {port: actual.count(port) for port in set(expected)}
-                expected_counts = {port: expected.count(port) for port in (expected)}
-                differences = {
-                    port: (expected_counts[port] - actual_counts.get(port, 0))
-                    for port in expected_counts
-                }
-                candidates = {
-                    port: deficit
-                    for port, deficit in differences.items()
-                    if deficit > 0
-                }
-                if candidates:
-                    output_port = max(candidates, key=candidates.get)
+            output_port = sname[llink["fromPort"]]
+            input_port = tname[llink["toPort"]]
             sdrop.addOutput(tdrop, name=output_port)
-            tdrop.addProducer(sdrop, name=tname)
+            tdrop.addProducer(sdrop, name=input_port)
+            if "port_map" not in tdrop:
+                tdrop["port_map"] = {input_port:output_port}
+            else:
+                tdrop["port_map"][input_port] = output_port
+
             if Categories.BASH_SHELL_APP == s_type:
                 bc = src_drop["command"]
                 bc.add_output_param(tlgn.id, tgt_drop["oid"])
@@ -533,6 +522,7 @@ def _link_drops(
                 self._gather_cache[gather_oid][2].append(tgt_drop)
             else:  # sdrop is a data drop
                 # there should be only one port, get the name
+                # ^ TODO This comment is no longer true, need to address
                 portId = llink["fromPort"] if "fromPort" in llink else None
                 sname = slgn._getPortName("outputPorts", portId=portId)
                 # could be multiple ports, need to identify
@@ -662,9 +652,6 @@ def unroll_to_tpl(self):
                             )
                         )
                     # first add the outer construct (scatter, gather, group-by) boundary
-                    # oc = slgn.group.group
-                    # if (oc is not None and (not oc.is_loop())):
-                    #     pass
                     loop_chunk_size = slgn.group.dop
                     for i, chunk in enumerate(
                         self._split_list(sdrops, loop_chunk_size)
@@ -678,10 +665,6 @@ def unroll_to_tpl(self):
                                     tdrops[i * loop_chunk_size + j + 1],
                                     lk,
                                 )
-
-                    # for i, sdrop in enumerate(sdrops):
-                    #     if (i < lsd - 1):
-                    #         self._link_drops(slgn, tlgn, sdrop, tdrops[i + 1])
                 elif (
                     slgn.group is not None
                     and slgn.group.is_loop
@@ -820,7 +803,6 @@ def unroll_to_tpl(self):
                 else:
                     data_drop.addConsumer(output_drop, name=sname)
                     output_drop.addInput(data_drop, name=sname)
-                # print(data_drop['nm'], data_drop['oid'], '-->', output_drop['nm'], output_drop['oid'])
 
         logger.info(
             "Unroll progress - %d links done for session %s",
@@ -841,16 +823,11 @@ def unroll_to_tpl(self):
                     if "grp-data_drop" in sl_drop:
                         del sl_drop["grp-data_drop"]
             elif lgn.is_gather:
-                # lid_sub = "{0}-gather-data".format(lid)
                 del self._drop_dict[lid]
-                # for sl_drop in self._drop_dict[lid]:
-                #     if 'gather-data_drop' in sl_drop:
-                #         del sl_drop['gather-data_drop']
             elif lgn.is_subgraph:
                 # Remove the SubGraph construct drop
                 if lgn.jd["isSubGraphConstruct"]:
                     del self._drop_dict[lid]
-
                 else:
                     pass
 
diff --git a/daliuge-translator/dlg/dropmake/lg_node.py b/daliuge-translator/dlg/dropmake/lg_node.py
index 8e4457559..4e4d64cb3 100644
--- a/daliuge-translator/dlg/dropmake/lg_node.py
+++ b/daliuge-translator/dlg/dropmake/lg_node.py
@@ -748,10 +748,8 @@ def _getPortName(self, ports: str = "outputPorts", index: int = 0, portId=None):
                     if portId is None or field["id"] == portId:
                         name = field["name"]
                     # can't be sure that name is unique
-                    if name not in ports_dict:
-                        ports_dict[name] = [field["id"]]
-                    else:
-                        ports_dict[name].append(field["id"])
+                    if field["id"] not in ports_dict:
+                        ports_dict[field["id"]] = name
         return name if index >= 0 else ports_dict
 
     def _create_groupby_drops(self, drop_spec):
diff --git a/daliuge-translator/setup.py b/daliuge-translator/setup.py
index 6a74dae2b..33bc6ff67 100644
--- a/daliuge-translator/setup.py
+++ b/daliuge-translator/setup.py
@@ -130,7 +130,8 @@ def package_files(directory):
 extra_requires = {
     "test": [
         "pytest",
-        "eagle-test-graphs",
+        "eagle-test-graphs @ git+https://github.com/ICRAR/EAGLE_test_repo@LIU-5",
+        # "eagle-test-graphs==0.1.6",
         "np-merklelib",
         "parameterized>=0.9.0",
         "ruamel.yaml==0.16.0",
diff --git a/daliuge-translator/test/dropmake/test_lg.py b/daliuge-translator/test/dropmake/test_lg.py
index 9fec0c206..752b2d320 100644
--- a/daliuge-translator/test/dropmake/test_lg.py
+++ b/daliuge-translator/test/dropmake/test_lg.py
@@ -20,12 +20,10 @@
 #    MA 02111-1307  USA
 #
 
-import json
 import pickle
 import unittest
 
-from dlg.common import CategoryType
-from dlg.dropmake import path_utils
+from dlg.common import CategoryType, path_utils
 from dlg.dropmake.lg import LG
 
 NODES = 'nodeDataArray'
diff --git a/daliuge-translator/test/dropmake/test_pg_gen.py b/daliuge-translator/test/dropmake/test_pg_gen.py
index 4f36bad2e..aba4caa76 100644
--- a/daliuge-translator/test/dropmake/test_pg_gen.py
+++ b/daliuge-translator/test/dropmake/test_pg_gen.py
@@ -28,7 +28,7 @@
 from dlg.dropmake.lg import LG
 from dlg.dropmake.pgt import PGT, GPGTNoNeedMergeException
 from dlg.dropmake.pgtp import MetisPGTP, MySarkarPGTP, MinNumPartsPGTP
-from dlg.dropmake import path_utils
+from dlg.common import path_utils
 
 """
 python -m unittest test.dropmake.test_pg_gen
diff --git a/daliuge-translator/test/dropmake/test_scheduler.py b/daliuge-translator/test/dropmake/test_scheduler.py
index 7af537e2e..5f97f8e7f 100644
--- a/daliuge-translator/test/dropmake/test_scheduler.py
+++ b/daliuge-translator/test/dropmake/test_scheduler.py
@@ -22,7 +22,6 @@
 import os
 import unittest
 
-import pkg_resources
 import psutil
 from dlg.dropmake.lg import LG
 from dlg.dropmake.scheduler import (
@@ -34,7 +33,7 @@
     PSOScheduler,
 )
 
-from dlg.dropmake.path_utils import get_lg_fpath
+from dlg.common.path_utils import get_lg_fpath
 
 if "DALIUGE_TESTS_RUNLONGTESTS" in os.environ:
     skip_long_tests = not bool(os.environ["DALIUGE_TESTS_RUNLONGTESTS"])
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 8c5c381c5..58bdf4691 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -7,4 +7,4 @@ myst-parser
 ipython
 np-merklelib >= 1.0
 gputil >= 1.4.0
-python-benedict[all]
\ No newline at end of file
+python-benedict[all]


Reviews:
user: login: sourcery-ai[bot]
body: Hey @myxie - I've reviewed your changes - here's some feedback:

**Overall Comments**:
- The changes to `_ports2args` look quite complex - can you add some comments to explain the logic?
- The addition of the `Argument` dataclass is a good way to manage arguments, but it's only used in `named_port_utils.py` - can you move it to be local to that file?

<details>
<summary>Here's what I looked at during the review</summary>

- 🟡 **General issues**: 2 issues found
- 🟢 **Security**: all looks good
- 🟢 **Testing**: all looks good
- 🟡 **Complexity**: 1 issue found
- 🟢 **Documentation**: all looks good
</details>

***

<details>
<summary>Sourcery is free for open source - if you like our reviews please consider sharing them ✨</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20got%20an%20instant%20code%20review%20from%20%40SourceryAI%2C%20and%20it%20was%20brilliant%21%20It%27s%20free%20for%20open%20source%20and%20has%20a%20free%20trial%20for%20private%20code.%20Check%20it%20out%20https%3A//sourcery.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20got%20an%20instant%20code%20review%20from%20%40SourceryAI%2C%20and%20it%20was%20brilliant%21%20It%27s%20free%20for%20open%20source%20and%20has%20a%20free%20trial%20for%20private%20code.%20Check%20it%20out%20https%3A//sourcery.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https://sourcery.ai)
- [Facebook](https://www.facebook.com/sharer/sharer.php?u=https://sourcery.ai)

</details>

<sub>
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T01:59:03Z
user: login: myxie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T02:29:15Z
user: login: myxie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T02:34:54Z
user: login: myxie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T02:36:05Z
user: login: sourcery-ai[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T02:36:16Z
user: login: myxie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T03:04:04Z
user: login: myxie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T03:05:13Z
user: login: awicenec
body: Happy to approve this now. It is working for me, thus we'll leave eventual issues to actual use cases.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T06:44:37Z

Review comments:
diff_hunk: @@ -252,13 +252,21 @@ class PyFuncApp(BarrierAppDROP):
 
     func_name = dlg_string_param("func_name", None)
     # func_code = dlg_bytes_param("func_code", None) # bytes or base64 string
-    input_parser: DropParser = dlg_enum_param(DropParser, "input_parser", DropParser.PICKLE)  # type: ignore
-    output_parser: DropParser = dlg_enum_param(DropParser, "output_parser", DropParser.PICKLE)  # type: ignore
+    input_parser: DropParser = dlg_enum_param(DropParser, "input_parser",
+                                              DropParser.PICKLE)  # type: ignore
+    output_parser: DropParser = dlg_enum_param(DropParser, "output_parser",
+                                               DropParser.PICKLE)  # type: ignore
     func_arg_mapping = dlg_dict_param("func_arg_mapping", {})
     func_defaults = dlg_dict_param("func_defaults", {})
     func: Callable
     fdefaults: dict
 
+    def __repr__(self):
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **issue (bug_risk):** __repr__ should always return a string.

In the '__repr__' method, the else block returns 'self', which may not be a string. It is advisable to ensure that '__repr__' consistently returns a string to prevent unexpected behavior.
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,26 +373,23 @@ def _map_input_ports_to_params(self):
                     continue
                 portName = param[key]
                 portValue = ""
-                producerUid = p.uid
                 if portName in p.parameters:
                     portValue = p.parameters[param[key]]
-                # TODO This currently only allows 1 UID -> Portname/Value
-                # Investigate UID -> [Portname1:Value1, Portnam2:value2,..,]
-                producerPortValueMap[producerUid] = {"portname": portName,
-                                                     "value": portValue}
+                producerPortValueMap[producerUid][portName] = portValue
 
         for port in dropInputPorts:
             try:
                 port.items()
             except AttributeError:
                 logging.debug("Producer %s does not have named ports", p.uid)
                 continue
-            for uid, portname in port.items():
+            for uid, input_port_name in port.items():
                 try:
-                    print(uid, portname)
-                    tmp = producerPortValueMap[uid]
-                    if tmp['portname'] == portname:
-                        finalDropPortMap[portname] = tmp['value']
+                    ouput_port_name = self.parameters["port_map"][input_port_name]
+                    print(uid, input_port_name)
path: daliuge-engine/dlg/data/drops/data_base.py
user: login: sourcery-ai[bot]
body: **issue:** Remove debugging print statements.

There is a 'print' call left in the production code. It is recommended to use logging (e.g. logging.debug) instead so that output can be controlled by log level and to avoid unintended console output.
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **issue (complexity):** Consider refactoring the large, nested functions `_populate_arguments_with_graph_data` and `_map_parameters_to_func_args` into smaller, dedicated helper functions to improve readability and reduce complexity by extracting the value evaluation and file path update logic into separate helper functions, respectively, and then calling these helpers from the main functions.

```markdown
Consider splitting some of the responsibilities from the two large, highly nested functions into smaller, dedicated helpers. For example, in `_populate_arguments_with_graph_data` you can separate the “appArgs value evaluation” logic from the mapping of arguments, and in `_map_parameters_to_func_args` isolate the file path update. This will reduce nesting and improve readability.

For example, extract the value evaluation into its own helper:

--------------------------------------------------
def _evaluate_arg_value(self, arg):
    """Evaluate the value if it is a JSON/Complex type."""
    value = self._applicationArgs[arg]["value"]
    encoding = self._applicationArgs[arg].get("encoding")
    try:
        if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
            value = ast.literal_eval(value)
    except (ValueError, SyntaxError):
        logger.error("Unable to evaluate %s", self._applicationArgs[arg]["value"])
    return value, encoding
--------------------------------------------------

Then use it inside `_populate_arguments_with_graph_data`:

--------------------------------------------------
for arg in self._applicationArgs:
    value, encoding = self._evaluate_arg_value(arg)
    for arg_map in [positionalArgsMap, keywordArgsMap]:
         if arg in arg_map:
             argument = arg_map[arg]
             # update only if new values are truthy
             argument.value = value if value else argument.value
             argument.encoding = encoding if encoding else argument.encoding
             arg_map[arg] = argument
    if self._applicationArgs[arg].get("usage") in ["InputOutput", "OutputPort"]:
         input_outputs.append(arg)
--------------------------------------------------

Similarly, extract the file path update logic into its own helper:

--------------------------------------------------
def _update_argument_filepath(self, arg, arg_map):
    output_uid = self.attr_uid_map.get(arg)
    output_drop = next((drop for drop in self.outputs if drop.uid == output_uid), None)
    if output_drop:
         argument = arg_map[arg]
         if DropParser(argument.encoding) == DropParser.PATH:
             argument.value = filepath_from_string(argument.value,
                                                   uid=output_drop.uid,
                                                   humanKey=output_drop._humanKey)
         arg_map[arg] = argument
    return arg_map
--------------------------------------------------

Then, in `_update_filepaths`, loop over both maps:

--------------------------------------------------
for arg_map in [keywordArgsMap, positionalArgsMap]:
    if arg in arg_map:
         arg_map = self._update_argument_filepath(arg, arg_map)
--------------------------------------------------

These focused helpers reduce the cognitive load in your main functions while keeping the original functionality intact.
```
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Merge nested if conditions ([`merge-nested-ifs`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/merge-nested-ifs))

```suggestion
            if kw in self._applicationArgs and isinstance(self._applicationArgs[kw]["value"],
                                          bool or self._applicationArgs[kw]["value"] or
                                          self._applicationArgs[kw]["precious"], ):
                self._applicationArgs.pop(kw)

```

<br/><details><summary>Explanation</summary>Too much nesting can make code difficult to understand, and this is especially
true in Python, where there are no brackets to help out with the delineation of
different nesting levels.

Reading deeply nested code is confusing, since you have to keep track of which
conditions relate to which levels. We therefore strive to reduce nesting where
possible, and the situation where two `if` conditions can be combined using
`and` is an easy win.
</details>
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
 
-    def _initialise_positional_args(self, argsDict: dict):
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
 
         for arg, value in argsDict.items():
-            argsDict[arg] = {"value": value, "encoding": None}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
-                argsDict[arg]["value"] = param_arg
-
+                argsDict[arg].value = param_arg
+                argsDict[arg].encoding = "dill"
         return argsDict
 
     def _get_arg_info(self, arg):
-
+        """
+        Get necessary information for populating the argument
+        """
         value = self._applicationArgs[arg]["value"]
-        encoding = self._applicationArgs[arg].get("encoding", "dill")
+        encoding = self._applicationArgs[arg].get("encoding")
+        precious = self._applicationArgs[arg].get("precious")
+        positional = self._applicationArgs[arg].get("positional")
+        portType = self._applicationArgs[arg].get("usage")
 
-        return value, encoding
+        return value, encoding, precious, positional, portType
 
-    def _init_appArgs(self, pargsDict: dict, keyargsDict: dict, posargs: list) -> list:
+    def _populate_arguments_with_graph_data(self, positionalArgsMap, keywordArgsMap):
         """
-        Identify and fill application arguments.
+         Go through the parameters that were provided by the user to the Logical Graph and
+        populate the application argument with the data.
 
-        Deals with positional and keyword arguments.
+        If the user didn't provide any input, or the data is expected to come through an
+        InputPort, the app argument will not be updated.
 
-        pargsDict: dict, 
+        :param positionalArgsMap:
+        :param keywordArgsMap:
 
-        keyargsDict: 
+        :return: positionalArgsMap,
+        :return: keywordArgsMap
+        :return: input_outputs
+        """
 
-        posargs: list 
+        vparg = []
+        vkarg = {}
+        input_outputs = []
+
+        # Legacy, used for graphs reliant on input/output parser
+        if not self._applicationArgs:
+            encoding = self.input_parser or DropParser.DILL
+            for key in positionalArgsMap:
+                positionalArgsMap[key].encoding = encoding
+            logger.debug("AppArgs/pargsDict: %s", positionalArgsMap)
+
+            return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
+
+        # If we overwrite any defaults with what is store in AppArgs "value"
+        for arg in self._applicationArgs:
+            # check value type and interpret
+            value, encoding, precious, positional, portType = self._get_arg_info(
+                arg)
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                try:
+                    value = ast.literal_eval(value)
+                    logger.debug(f"Evaluated %s to %s",
+                                 value, type(value))
+                    self._applicationArgs[arg].value = value
+                except ValueError:
+                    logger.error("Unable to evaluate %s",
+                                 self._applicationArgs[arg]["value"])
+            for arg_map in [positionalArgsMap, keywordArgsMap]:
+                if arg in arg_map:
+                    argument = arg_map[arg]
+                    argument.value = value if value else argument.value
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace if-expression with `or` ([`or-if-exp-identity`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/or-if-exp-identity))

```suggestion
                    argument.value = value or argument.value
```

<br/><details><summary>Explanation</summary>Here we find ourselves setting a value if it evaluates to `True`, and otherwise
using a default.

The 'After' case is a bit easier to read and avoids the duplication of
`input_currency`.

It works because the left-hand side is evaluated first. If it evaluates to
true then `currency` will be set to this and the right-hand side will not be
evaluated. If it evaluates to false the right-hand side will be evaluated and
`currency` will be set to `DEFAULT_CURRENCY`.
</details>
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
 
-    def _initialise_positional_args(self, argsDict: dict):
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
 
         for arg, value in argsDict.items():
-            argsDict[arg] = {"value": value, "encoding": None}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
-                argsDict[arg]["value"] = param_arg
-
+                argsDict[arg].value = param_arg
+                argsDict[arg].encoding = "dill"
         return argsDict
 
     def _get_arg_info(self, arg):
-
+        """
+        Get necessary information for populating the argument
+        """
         value = self._applicationArgs[arg]["value"]
-        encoding = self._applicationArgs[arg].get("encoding", "dill")
+        encoding = self._applicationArgs[arg].get("encoding")
+        precious = self._applicationArgs[arg].get("precious")
+        positional = self._applicationArgs[arg].get("positional")
+        portType = self._applicationArgs[arg].get("usage")
 
-        return value, encoding
+        return value, encoding, precious, positional, portType
 
-    def _init_appArgs(self, pargsDict: dict, keyargsDict: dict, posargs: list) -> list:
+    def _populate_arguments_with_graph_data(self, positionalArgsMap, keywordArgsMap):
         """
-        Identify and fill application arguments.
+         Go through the parameters that were provided by the user to the Logical Graph and
+        populate the application argument with the data.
 
-        Deals with positional and keyword arguments.
+        If the user didn't provide any input, or the data is expected to come through an
+        InputPort, the app argument will not be updated.
 
-        pargsDict: dict, 
+        :param positionalArgsMap:
+        :param keywordArgsMap:
 
-        keyargsDict: 
+        :return: positionalArgsMap,
+        :return: keywordArgsMap
+        :return: input_outputs
+        """
 
-        posargs: list 
+        vparg = []
+        vkarg = {}
+        input_outputs = []
+
+        # Legacy, used for graphs reliant on input/output parser
+        if not self._applicationArgs:
+            encoding = self.input_parser or DropParser.DILL
+            for key in positionalArgsMap:
+                positionalArgsMap[key].encoding = encoding
+            logger.debug("AppArgs/pargsDict: %s", positionalArgsMap)
+
+            return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
+
+        # If we overwrite any defaults with what is store in AppArgs "value"
+        for arg in self._applicationArgs:
+            # check value type and interpret
+            value, encoding, precious, positional, portType = self._get_arg_info(
+                arg)
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                try:
+                    value = ast.literal_eval(value)
+                    logger.debug(f"Evaluated %s to %s",
+                                 value, type(value))
+                    self._applicationArgs[arg].value = value
+                except ValueError:
+                    logger.error("Unable to evaluate %s",
+                                 self._applicationArgs[arg]["value"])
+            for arg_map in [positionalArgsMap, keywordArgsMap]:
+                if arg in arg_map:
+                    argument = arg_map[arg]
+                    argument.value = value if value else argument.value
+                    argument.encoding = encoding if encoding else argument.encoding
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace if-expression with `or` ([`or-if-exp-identity`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/or-if-exp-identity))

```suggestion
                    argument.encoding = encoding or argument.encoding
```

<br/><details><summary>Explanation</summary>Here we find ourselves setting a value if it evaluates to `True`, and otherwise
using a default.

The 'After' case is a bit easier to read and avoids the duplication of
`input_currency`.

It works because the left-hand side is evaluated first. If it evaluates to
true then `currency` will be set to this and the right-hand side will not be
evaluated. If it evaluates to false the right-hand side will be evaluated and
`currency` will be set to `DEFAULT_CURRENCY`.
</details>
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
 
-    def _initialise_positional_args(self, argsDict: dict):
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
 
         for arg, value in argsDict.items():
-            argsDict[arg] = {"value": value, "encoding": None}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
-                argsDict[arg]["value"] = param_arg
-
+                argsDict[arg].value = param_arg
+                argsDict[arg].encoding = "dill"
         return argsDict
 
     def _get_arg_info(self, arg):
-
+        """
+        Get necessary information for populating the argument
+        """
         value = self._applicationArgs[arg]["value"]
-        encoding = self._applicationArgs[arg].get("encoding", "dill")
+        encoding = self._applicationArgs[arg].get("encoding")
+        precious = self._applicationArgs[arg].get("precious")
+        positional = self._applicationArgs[arg].get("positional")
+        portType = self._applicationArgs[arg].get("usage")
 
-        return value, encoding
+        return value, encoding, precious, positional, portType
 
-    def _init_appArgs(self, pargsDict: dict, keyargsDict: dict, posargs: list) -> list:
+    def _populate_arguments_with_graph_data(self, positionalArgsMap, keywordArgsMap):
         """
-        Identify and fill application arguments.
+         Go through the parameters that were provided by the user to the Logical Graph and
+        populate the application argument with the data.
 
-        Deals with positional and keyword arguments.
+        If the user didn't provide any input, or the data is expected to come through an
+        InputPort, the app argument will not be updated.
 
-        pargsDict: dict, 
+        :param positionalArgsMap:
+        :param keywordArgsMap:
 
-        keyargsDict: 
+        :return: positionalArgsMap,
+        :return: keywordArgsMap
+        :return: input_outputs
+        """
 
-        posargs: list 
+        vparg = []
+        vkarg = {}
+        input_outputs = []
+
+        # Legacy, used for graphs reliant on input/output parser
+        if not self._applicationArgs:
+            encoding = self.input_parser or DropParser.DILL
+            for key in positionalArgsMap:
+                positionalArgsMap[key].encoding = encoding
+            logger.debug("AppArgs/pargsDict: %s", positionalArgsMap)
+
+            return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
+
+        # If we overwrite any defaults with what is store in AppArgs "value"
+        for arg in self._applicationArgs:
+            # check value type and interpret
+            value, encoding, precious, positional, portType = self._get_arg_info(
+                arg)
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                try:
+                    value = ast.literal_eval(value)
+                    logger.debug(f"Evaluated %s to %s",
+                                 value, type(value))
+                    self._applicationArgs[arg].value = value
+                except ValueError:
+                    logger.error("Unable to evaluate %s",
+                                 self._applicationArgs[arg]["value"])
+            for arg_map in [positionalArgsMap, keywordArgsMap]:
+                if arg in arg_map:
+                    argument = arg_map[arg]
+                    argument.value = value if value else argument.value
+                    argument.encoding = encoding if encoding else argument.encoding
+                    argument.precious = precious if precious else argument.precious
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace if-expression with `or` ([`or-if-exp-identity`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/or-if-exp-identity))

```suggestion
                    argument.precious = precious or argument.precious
```

<br/><details><summary>Explanation</summary>Here we find ourselves setting a value if it evaluates to `True`, and otherwise
using a default.

The 'After' case is a bit easier to read and avoids the duplication of
`input_currency`.

It works because the left-hand side is evaluated first. If it evaluates to
true then `currency` will be set to this and the right-hand side will not be
evaluated. If it evaluates to false the right-hand side will be evaluated and
`currency` will be set to `DEFAULT_CURRENCY`.
</details>
created_at: 2025-04-09T01:59:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
 
-    def _initialise_positional_args(self, argsDict: dict):
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
 
         for arg, value in argsDict.items():
-            argsDict[arg] = {"value": value, "encoding": None}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
-                argsDict[arg]["value"] = param_arg
-
+                argsDict[arg].value = param_arg
+                argsDict[arg].encoding = "dill"
         return argsDict
 
     def _get_arg_info(self, arg):
-
+        """
+        Get necessary information for populating the argument
+        """
         value = self._applicationArgs[arg]["value"]
-        encoding = self._applicationArgs[arg].get("encoding", "dill")
+        encoding = self._applicationArgs[arg].get("encoding")
+        precious = self._applicationArgs[arg].get("precious")
+        positional = self._applicationArgs[arg].get("positional")
+        portType = self._applicationArgs[arg].get("usage")
 
-        return value, encoding
+        return value, encoding, precious, positional, portType
 
-    def _init_appArgs(self, pargsDict: dict, keyargsDict: dict, posargs: list) -> list:
+    def _populate_arguments_with_graph_data(self, positionalArgsMap, keywordArgsMap):
         """
-        Identify and fill application arguments.
+         Go through the parameters that were provided by the user to the Logical Graph and
+        populate the application argument with the data.
 
-        Deals with positional and keyword arguments.
+        If the user didn't provide any input, or the data is expected to come through an
+        InputPort, the app argument will not be updated.
 
-        pargsDict: dict, 
+        :param positionalArgsMap:
+        :param keywordArgsMap:
 
-        keyargsDict: 
+        :return: positionalArgsMap,
+        :return: keywordArgsMap
+        :return: input_outputs
+        """
 
-        posargs: list 
+        vparg = []
+        vkarg = {}
+        input_outputs = []
+
+        # Legacy, used for graphs reliant on input/output parser
+        if not self._applicationArgs:
+            encoding = self.input_parser or DropParser.DILL
+            for key in positionalArgsMap:
+                positionalArgsMap[key].encoding = encoding
+            logger.debug("AppArgs/pargsDict: %s", positionalArgsMap)
+
+            return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
+
+        # If we overwrite any defaults with what is store in AppArgs "value"
+        for arg in self._applicationArgs:
+            # check value type and interpret
+            value, encoding, precious, positional, portType = self._get_arg_info(
+                arg)
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                try:
+                    value = ast.literal_eval(value)
+                    logger.debug(f"Evaluated %s to %s",
+                                 value, type(value))
+                    self._applicationArgs[arg].value = value
+                except ValueError:
+                    logger.error("Unable to evaluate %s",
+                                 self._applicationArgs[arg]["value"])
+            for arg_map in [positionalArgsMap, keywordArgsMap]:
+                if arg in arg_map:
+                    argument = arg_map[arg]
+                    argument.value = value if value else argument.value
+                    argument.encoding = encoding if encoding else argument.encoding
+                    argument.precious = precious if precious else argument.precious
+                    argument.positional = positional if positional else argument.positional
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace if-expression with `or` ([`or-if-exp-identity`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/or-if-exp-identity))

```suggestion
                    argument.positional = positional or argument.positional
```

<br/><details><summary>Explanation</summary>Here we find ourselves setting a value if it evaluates to `True`, and otherwise
using a default.

The 'After' case is a bit easier to read and avoids the duplication of
`input_currency`.

It works because the left-hand side is evaluated first. If it evaluates to
true then `currency` will be set to this and the right-hand side will not be
evaluated. If it evaluates to false the right-hand side will be evaluated and
`currency` will be set to `DEFAULT_CURRENCY`.
</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,126 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+
+"""
+This module is used to test FileDROP input/output naming through named ports.
+"""
+import datetime
+import dill
+import pytest
+import unittest
+
+from importlib.resources import files
+
+from dlg.ddap_protocol import DROPStates
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
path: daliuge-engine/test/named_ports/test_filedrops.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** Don't import test modules. ([`dont-import-test-modules`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/dont-import-test-modules))

<details><summary>Explanation</summary>Don't import test modules.

Tests should be self-contained and don't depend on each other.

If a helper function is used by multiple tests,
define it in a helper module,
instead of importing one test from the other.
</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,126 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+
+"""
+This module is used to test FileDROP input/output naming through named ports.
+"""
+import datetime
+import dill
+import pytest
+import unittest
+
+from importlib.resources import files
+
+from dlg.ddap_protocol import DROPStates
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
+from daliuge_tests.engine import test_filedrops as test_graphs
+
+# Note this test will only run with a full installation of DALiuGE.
+pexpect = pytest.importorskip("dlg.dropmake")
+
+class TestBasicApp(unittest.TestCase):
+    """
+    Given a simple graph with:
+
+    <return_path("test_{datetime}.txt")> --> <FileDrop>
+
+    Where:
+        >>> def return_path(fn):
+        >>>     return fn
+
+    Make sure that it runs and produces a graph with name test_<current_date>.txt, and
+    that the content is also test_<current_date>.txt
+
+    """
+
+    graph = f"{files(test_graphs)}/Filepath_testPG.graph"
+
+    def test_pyfunc_to_fileapp(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
path: daliuge-engine/test/named_ports/test_filedrops.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** Avoid loops in tests. ([`no-loop-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-loop-in-tests))

<details><summary>Explanation</summary>Avoid complex code, like loops, in test functions.

Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals

Some ways to fix this:

* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.

> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.

Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,126 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+
+"""
+This module is used to test FileDROP input/output naming through named ports.
+"""
+import datetime
+import dill
+import pytest
+import unittest
+
+from importlib.resources import files
+
+from dlg.ddap_protocol import DROPStates
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
+from daliuge_tests.engine import test_filedrops as test_graphs
+
+# Note this test will only run with a full installation of DALiuGE.
+pexpect = pytest.importorskip("dlg.dropmake")
+
+class TestBasicApp(unittest.TestCase):
+    """
+    Given a simple graph with:
+
+    <return_path("test_{datetime}.txt")> --> <FileDrop>
+
+    Where:
+        >>> def return_path(fn):
+        >>>     return fn
+
+    Make sure that it runs and produces a graph with name test_<current_date>.txt, and
+    that the content is also test_<current_date>.txt
+
+    """
+
+    graph = f"{files(test_graphs)}/Filepath_testPG.graph"
+
+    def test_pyfunc_to_fileapp(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
+
+        dt = datetime.date.today().strftime("%Y-%m-%d")
+        # Leaf Node 1 has been encoded first as numpy, second as UTF-8.
+        leaf = leafs.pop()
+        self.assertEqual(f"test_{dt}.txt", leaf.filename)
+        desc = leaf.open()
+        self.assertEqual(f"test_{dt}.txt", leaf.read(desc).decode())
+
+
+class TestBasicAppWithMemoryDropFileInput(unittest.TestCase):
+    """
+    This is the same as the TestBasicApp graph, except we use a memory drop with Pydata
+    that defines the path name instead.
+    
+    <Pydata("test_{datetime}.txt")> --> <return_path> --> <FileDROP>
+    """
+
+    graph = f"{files(test_graphs)}/Filepath_test_InputOutputPG.graph"
+
+    def test_pyfuncapp_to_file(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False, app_root=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
path: daliuge-engine/test/named_ports/test_filedrops.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** Avoid loops in tests. ([`no-loop-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-loop-in-tests))

<details><summary>Explanation</summary>Avoid complex code, like loops, in test functions.

Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals

Some ways to fix this:

* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.

> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.

Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,126 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+
+"""
+This module is used to test FileDROP input/output naming through named ports.
+"""
+import datetime
+import dill
+import pytest
+import unittest
+
+from importlib.resources import files
+
+from dlg.ddap_protocol import DROPStates
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
+from daliuge_tests.engine import test_filedrops as test_graphs
+
+# Note this test will only run with a full installation of DALiuGE.
+pexpect = pytest.importorskip("dlg.dropmake")
+
+class TestBasicApp(unittest.TestCase):
+    """
+    Given a simple graph with:
+
+    <return_path("test_{datetime}.txt")> --> <FileDrop>
+
+    Where:
+        >>> def return_path(fn):
+        >>>     return fn
+
+    Make sure that it runs and produces a graph with name test_<current_date>.txt, and
+    that the content is also test_<current_date>.txt
+
+    """
+
+    graph = f"{files(test_graphs)}/Filepath_testPG.graph"
+
+    def test_pyfunc_to_fileapp(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
+
+        dt = datetime.date.today().strftime("%Y-%m-%d")
+        # Leaf Node 1 has been encoded first as numpy, second as UTF-8.
+        leaf = leafs.pop()
+        self.assertEqual(f"test_{dt}.txt", leaf.filename)
+        desc = leaf.open()
+        self.assertEqual(f"test_{dt}.txt", leaf.read(desc).decode())
+
+
+class TestBasicAppWithMemoryDropFileInput(unittest.TestCase):
+    """
+    This is the same as the TestBasicApp graph, except we use a memory drop with Pydata
+    that defines the path name instead.
+    
+    <Pydata("test_{datetime}.txt")> --> <return_path> --> <FileDROP>
+    """
+
+    graph = f"{files(test_graphs)}/Filepath_test_InputOutputPG.graph"
+
+    def test_pyfuncapp_to_file(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False, app_root=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
+
+        dt = datetime.date.today().strftime("%Y-%m-%d")
+        # Leaf Node 1 has been encoded first as numpy, second as UTF-8.
+        leaf = leafs.pop()
+        self.assertEqual(f"test_{dt}.txt", leaf.filename)
+        desc = leaf.open()
+        self.assertEqual(f"test_{dt}.txt", leaf.read(desc).decode())
+
+
+class TestAppFileChain(unittest.TestCase):
+    """
+    This is an extension of TestBasicApp graph, where we now read in the output and use
+    the path of the first FileDROP, and do the same thing again whilst modifying the
+    filename and data in the file.
+
+    This shows that we can re-use data using the user-supplied filename approach.
+
+    """
+    graph = f"{files(test_graphs)}/Filepath_test_multiple_appsPG.graph"
+
+    def test_pyfunc_to_file_with_multiple_apps(self):
+        """
+        Confirm filename and file output is correct
+        """
+        leafs = create_and_run_graph_spec_from_graph_file(
+            self, self.graph, resolve_path=False)
+        for l in leafs:
+            self.assertEqual(DROPStates.COMPLETED, l.status)
path: daliuge-engine/test/named_ports/test_filedrops.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** Avoid loops in tests. ([`no-loop-in-tests`](https://docs.sourcery.ai/Reference/Rules-and-In-Line-Suggestions/Python/Default-Rules/no-loop-in-tests))

<details><summary>Explanation</summary>Avoid complex code, like loops, in test functions.

Google's software engineering guidelines says:
"Clear tests are trivially correct upon inspection"
To reach that avoid complex code in tests:
* loops
* conditionals

Some ways to fix this:

* Use parametrized tests to get rid of the loop.
* Move the complex logic into helpers.
* Move the complex part into pytest fixtures.

> Complexity is most often introduced in the form of logic. Logic is defined via the imperative parts of programming languages such as operators, loops, and conditionals. When a piece of code contains logic, you need to do a bit of mental computation to determine its result instead of just reading it off of the screen. It doesn't take much logic to make a test more difficult to reason about.

Software Engineering at Google / [Don't Put Logic in Tests](https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests)
</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -75,12 +75,12 @@
     if a.defaults:
         fdefaults["kwargs"] = dict(
             zip(
-                a.args[-len(a.defaults) :],
+                a.args[-len(a.defaults):],
                 [serialize_data(d) for d in a.defaults],
             )
         )
         adefaults["kwargs"] = dict(
-            zip(a.args[-len(a.defaults) :], [d for d in a.defaults])
+            zip(a.args[-len(a.defaults):], [d for d in a.defaults])
         )
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace identity comprehension with call to collection constructor ([`identity-comprehension`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/identity-comprehension/))

```suggestion
        adefaults["kwargs"] = dict(zip(a.args[-len(a.defaults):], list(a.defaults)))
```

<br/><details><summary>Explanation</summary>Convert list/set/tuple comprehensions that do not change the input elements into.

#### Before

```python
# List comprehensions
[item for item in coll]
[item for item in friends.names()]

# Dict comprehensions
{k: v for k, v in coll}
{k: v for k, v in coll.items()}  # Only if we know coll is a `dict`

# Unneeded call to `.items()`
dict(coll.items())  # Only if we know coll is a `dict`

# Set comprehensions
{item for item in coll}
```

#### After

```python
# List comprehensions
list(iter(coll))
list(iter(friends.names()))

# Dict comprehensions
dict(coll)
dict(coll)

# Unneeded call to `.items()`
dict(coll)

# Set comprehensions
set(coll)
```

All these comprehensions are just creating a copy of the original collection.
They can all be simplified by simply constructing a new collection directly. The
resulting code is easier to read and shows the intent more clearly.</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
 
-    def _initialise_positional_args(self, argsDict: dict):
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
 
         for arg, value in argsDict.items():
-            argsDict[arg] = {"value": value, "encoding": None}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Use named expression to simplify assignment and conditional ([`use-named-expression`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/use-named-expression/))

```suggestion
            if param_arg := self.parameters.get(arg):
```
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
 
-    def _initialise_positional_args(self, argsDict: dict):
+    def _initialise_args(self, argsDict: dict):
         """
         Ensure default arguments are properly setup.
         # If there are default values in the function sig
         # If there are values in the graph
         """
 
         for arg, value in argsDict.items():
-            argsDict[arg] = {"value": value, "encoding": None}
-            param_arg = self.parameters.get(arg, None)
+            argsDict[arg] = Argument(value)
+            param_arg = self.parameters.get(arg)
             if param_arg:
-                argsDict[arg]["value"] = param_arg
-
+                argsDict[arg].value = param_arg
+                argsDict[arg].encoding = "dill"
         return argsDict
 
     def _get_arg_info(self, arg):
-
+        """
+        Get necessary information for populating the argument
+        """
         value = self._applicationArgs[arg]["value"]
-        encoding = self._applicationArgs[arg].get("encoding", "dill")
+        encoding = self._applicationArgs[arg].get("encoding")
+        precious = self._applicationArgs[arg].get("precious")
+        positional = self._applicationArgs[arg].get("positional")
+        portType = self._applicationArgs[arg].get("usage")
 
-        return value, encoding
+        return value, encoding, precious, positional, portType
 
-    def _init_appArgs(self, pargsDict: dict, keyargsDict: dict, posargs: list) -> list:
+    def _populate_arguments_with_graph_data(self, positionalArgsMap, keywordArgsMap):
         """
-        Identify and fill application arguments.
+         Go through the parameters that were provided by the user to the Logical Graph and
+        populate the application argument with the data.
 
-        Deals with positional and keyword arguments.
+        If the user didn't provide any input, or the data is expected to come through an
+        InputPort, the app argument will not be updated.
 
-        pargsDict: dict, 
+        :param positionalArgsMap:
+        :param keywordArgsMap:
 
-        keyargsDict: 
+        :return: positionalArgsMap,
+        :return: keywordArgsMap
+        :return: input_outputs
+        """
 
-        posargs: list 
+        vparg = []
+        vkarg = {}
+        input_outputs = []
+
+        # Legacy, used for graphs reliant on input/output parser
+        if not self._applicationArgs:
+            encoding = self.input_parser or DropParser.DILL
+            for key in positionalArgsMap:
+                positionalArgsMap[key].encoding = encoding
+            logger.debug("AppArgs/pargsDict: %s", positionalArgsMap)
+
+            return positionalArgsMap, keywordArgsMap, input_outputs, vparg, vkarg
+
+        # If we overwrite any defaults with what is store in AppArgs "value"
+        for arg in self._applicationArgs:
+            # check value type and interpret
+            value, encoding, precious, positional, portType = self._get_arg_info(
+                arg)
+            if self._applicationArgs[arg]["type"] in ["Json", "Complex"]:
+                try:
+                    value = ast.literal_eval(value)
+                    logger.debug(f"Evaluated %s to %s",
+                                 value, type(value))
+                    self._applicationArgs[arg].value = value
+                except ValueError:
+                    logger.error("Unable to evaluate %s",
+                                 self._applicationArgs[arg]["value"])
+            for arg_map in [positionalArgsMap, keywordArgsMap]:
+                if arg in arg_map:
+                    argument = arg_map[arg]
+                    argument.value = value if value else argument.value
+                    argument.encoding = encoding if encoding else argument.encoding
+                    argument.precious = precious if precious else argument.precious
+                    argument.positional = positional if positional else argument.positional
+                    arg_map[arg] = argument
+
+            # if populate_maps_without_args:
+            #     positionalArgsMap[arg] = Argument(value=value, encoding=encoding,
+            #                                 precious=precious, positional=positional)
+            #
+            if portType == "InputOutput" or portType == "OutputPort":
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** We've found these issues:

- Replace multiple comparisons of same variable with `in` operator ([`merge-comparisons`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/merge-comparisons/))
- Low code quality found in PyFuncApp.\_populate\_arguments\_with\_graph\_data - 15% ([`low-code-quality`](https://docs.sourcery.ai/Reference/Default-Rules/comments/low-code-quality/))

```suggestion
            if portType in ["InputOutput", "OutputPort"]:
```

<br/><details><summary>Explanation</summary>
The quality score for this function is below the quality threshold of 25%.
This score is a combination of the method length, cognitive complexity and working memory.

How can you solve this?

It might be worth refactoring this function to make it shorter and more readable.

- Reduce the function length by extracting pieces of functionality out into
  their own functions. This is the most important thing you can do - ideally a
  function should be less than 10 lines.
- Reduce nesting, perhaps by introducing guard clauses to return early.
- Ensure that variables are tightly scoped, so that code using related concepts
  sits together within the function rather than being scattered.</details>
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -489,41 +619,44 @@
         """
         portargs = {}
         # 3. replace default argument values with named input ports
-        iitems = self._inputs
-        logger.debug("Mapping from _inputs: %s", iitems)
+        logger.debug("Mapping from _inputs: %s", self._inputs)
         logger.debug("Parameters: %s", self.parameters)
         if "inputs" in self.parameters and check_ports_dict(self.parameters["inputs"]):
             logger.debug("Mapping ports to inputs...")
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** We've found these issues:

- Extract code out into method ([`extract-method`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/extract-method/))
- Replace if statement with if expression ([`assign-if-exp`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/assign-if-exp/))
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -88,7 +88,14 @@ def string2json(string: str, pickle_flag: bool = False) -> list:
     """
     if not string:
         string = '""'
-    if not pickle_flag:
-        return json.loads(string)
-    else:
-        return pickle.dumps(json.loads(string))
+
+    try:
+        if not pickle_flag:
+            return json.loads(string)
+        else:
+            return pickle.dumps(json.loads(string))
+    except json.decoder.JSONDecodeError as e:
+        if not pickle_flag:
+            return []
+        else:
+            return pickle.dumps([])
path: daliuge-engine/dlg/apps/simple_functions.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace if statement with if expression ([`assign-if-exp`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/assign-if-exp/))

```suggestion
        return [] if not pickle_flag else pickle.dumps([])
```
created_at: 2025-04-09T01:59:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -116,10 +117,12 @@ def initialize(self, **kwargs):
 
     def _setupFilePaths(self):
         filepath = self.parameters.get("filepath", None)
path: daliuge-engine/dlg/data/drops/file.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** We've found these issues:

- Move setting of default value for variable into `else` branch ([`introduce-default-else`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/introduce-default-else/))
- Replace interpolated string formatting with f-string ([`replace-interpolation-with-fstring`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/replace-interpolation-with-fstring/))
- Replace if statement with if expression ([`assign-if-exp`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/assign-if-exp/))
created_at: 2025-04-09T01:59:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,145 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+"""
+Module with utility methods to create runtime file paths for the DALiuGE Engine.
+
+The Engine supports implicit filepath naming if FileDROPs have no user-defined name in
+the Logical Graph Template. This helps when multiple FileDROPs are created in a
+construct (e.g. Scatter or Loop) and we don't run the risk of overwriting them.
+
+This module provides the methods that support different filepath naming heuristics to
+facilitate implicit filepath naming at runtime.
+"""
+import datetime
+import logging
+import re
+import os
+import uuid
+
+NON_FILENAME_CHARACTERS = re.compile(r":|%s" % os.sep)
path: daliuge-engine/dlg/data/path_builder.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** Replace interpolated string formatting with f-string ([`replace-interpolation-with-fstring`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/replace-interpolation-with-fstring/))
created_at: 2025-04-09T01:59:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,145 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+"""
+Module with utility methods to create runtime file paths for the DALiuGE Engine.
+
+The Engine supports implicit filepath naming if FileDROPs have no user-defined name in
+the Logical Graph Template. This helps when multiple FileDROPs are created in a
+construct (e.g. Scatter or Loop) and we don't run the risk of overwriting them.
+
+This module provides the methods that support different filepath naming heuristics to
+facilitate implicit filepath naming at runtime.
+"""
+import datetime
+import logging
+import re
+import os
+import uuid
+
+NON_FILENAME_CHARACTERS = re.compile(r":|%s" % os.sep)
+
+def default_map():
+    """
+    Get the default map for the FSTRING replacement keywords
+    """
+    return {
+        "dlg": "DALIGUE",
+        "datetime": datetime.date.today().strftime("%Y-%m-%d"),
+        "uid": str(uuid.uuid4()),
+    } 
+
+def base_uid_filename(uid: str, humanKey: str):
+    """
+    This a basic filename generator, using the UID and humandReadableKey. The function
+    returns only the name of the file, and expects the full filepath to be handled by
+    the function caller.
+
+    Parameters
+    ----------
+    uid: str, the UID of the DROP
+    humanKey: str, a human-readable key (typically a truncated UID) for user-facing
+        display
+
+    Notes
+    -----
+    An example DROP UID would be:
+
+        '2024-10-30T12:01:57_0140555b-8c23-4d6a-9e24-e16c15555e8c_0'
+
+    This approach will lead to an abridged UID along with the humanKey appended:
+
+       '2024-10-30T12:01:57_0140555b_0/1/1'
+
+    In this case, 0/1/1 is the "humanKey", derived from the DALiuGE Translator
+    constructs (For more information on humanKey generation, refer to
+    dlg.common.truncateUidToKey).
+
+    The final part of the method replaces characters that cannot be in filenames from
+    the UIDs (e.g. "/").
+
+    Returns
+    -------
+        str: Base filename
+    """
+    if not uid or not humanKey:
+        return None
+    fn = uid.split("_")[0] + "_" + str(humanKey)
path: daliuge-engine/dlg/data/path_builder.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Remove unnecessary casts to int, str, float or bool ([`remove-unnecessary-cast`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/remove-unnecessary-cast/))

```suggestion
    fn = uid.split("_")[0] + "_" + humanKey
```
created_at: 2025-04-09T01:59:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -84,17 +117,29 @@ def serialize_applicationArgs(applicationArgs, prefix="--", separator=" "):
     logger.info("Constructed command line arguments: %s %s", pargs, kwargs)
     return (pargs, skwargs)
 
+def extract_encoded_value(args: dict, drop: "AbstractDROP", encoding:DropParser):
+    """
+
+    Parameters
+    ----------
+    args
+    drop
+    encoding
+
+    Returns
+    -------
+
+    """
 
 def identify_named_ports(
path: daliuge-engine/dlg/named_port_utils.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** We've found these issues:

- Use named expression to simplify assignment and conditional [×2] ([`use-named-expression`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/use-named-expression/))
- Add single value to dictionary directly rather than using update() [×2] ([`simplify-dictionary-update`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/simplify-dictionary-update/))
- Explicitly raise from a previous error ([`raise-from-previous-error`](https://docs.sourcery.ai/Reference/Default-Rules/suggestions/raise-from-previous-error/))
- Low code quality found in identify\_named\_ports - 24% ([`low-code-quality`](https://docs.sourcery.ai/Reference/Default-Rules/comments/low-code-quality/))

<br/><details><summary>Explanation</summary>




The quality score for this function is below the quality threshold of 25%.
This score is a combination of the method length, cognitive complexity and working memory.

How can you solve this?

It might be worth refactoring this function to make it shorter and more readable.

- Reduce the function length by extracting pieces of functionality out into
  their own functions. This is the most important thing you can do - ideally a
  function should be less than 10 lines.
- Reduce nesting, perhaps by introducing guard clauses to return early.
- Ensure that variables are tightly scoped, so that code using related concepts
  sits together within the function rather than being scattered.</details>
created_at: 2025-04-09T01:59:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,70 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+import unittest
+import datetime
+from dlg.data.path_builder import (find_dlg_fstrings, filepath_from_string,
+                                    base_uid_filename)
+
+
+class TestPathBuilders(unittest.TestCase):
+
+    def setUp(self):
+
+        self.filename = "test"
+
+
+    def testBaseUIDGenerator(self):
+        """
+        Using the baseUID Generator with the humanreadable key
+        """
+
+    def test_file_path_from_string(self):
+        uid = "123456"
+        res = filepath_from_string("prefix_{uid}_{datetime}.dat", uid=uid)
+        dstr = datetime.date.today().strftime("%Y-%m-%d")
+        self.assertEqual(f"prefix_123456_{dstr}.dat", res)
+        dt = "2025-08-17"
+        res = filepath_from_string("prefix_{uid}_{datetime}.dat",
+                                   uid=uid, datetime=dt)
+        self.assertEqual(f"prefix_123456_2025-08-17.dat", res)
path: daliuge-engine/test/named_ports/test_path_builder.py
user: login: sourcery-ai[bot]
body: **suggestion (code-quality):** Replace f-string with no interpolated values with string ([`remove-redundant-fstring`](https://docs.sourcery.ai/Reference/Default-Rules/refactorings/remove-redundant-fstring/))

```suggestion
        self.assertEqual("prefix_123456_2025-08-17.dat", res)
```
created_at: 2025-04-09T01:59:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -348,138 +370,246 @@
         logger.debug("Got signature for function %s %s", self.func, self.argsig)
         logger.debug("Got default values for arguments %s", self.arguments_defaults)
         logger.debug(f"initialized fn_defaults with {self.fn_defaults}")
+        return True
 
     def _clean_applicationArgs(self):
         """
         remove function definition arguments in applicationArgs
         """
         self.func_def_keywords = ["func_code", "func_name", "func_arg_mapping",
-            "input_parser", "output_parser", "func_defaults", "pickle", ]
+                                  "input_parser", "output_parser", "func_defaults",
+                                  "pickle", ]
 
         for kw in self.func_def_keywords:
             if kw in self._applicationArgs:  # these are the preferred ones now
                 if isinstance(self._applicationArgs[kw]["value"],
-                        bool or self._applicationArgs[kw]["value"] or
-                        self._applicationArgs[kw]["precious"], ):
+                              bool or self._applicationArgs[kw]["value"] or
+                              self._applicationArgs[kw]["precious"], ):
                     # only transfer if there is a value or precious is True
                     self._applicationArgs.pop(kw)
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: myxie
body: These methods already simplify things quite a lot, and I'd rather not change that area of the code even more. 
created_at: 2025-04-09T02:29:14Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,126 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+
+"""
+This module is used to test FileDROP input/output naming through named ports.
+"""
+import datetime
+import dill
+import pytest
+import unittest
+
+from importlib.resources import files
+
+from dlg.ddap_protocol import DROPStates
+from test.dlg_end_to_end_utils import create_and_run_graph_spec_from_graph_file
path: daliuge-engine/test/named_ports/test_filedrops.py
user: login: myxie
body: These are test helper utilities. 
created_at: 2025-04-09T02:34:54Z
author_association: COLLABORATOR
diff_hunk: @@ -75,12 +75,12 @@
     if a.defaults:
         fdefaults["kwargs"] = dict(
             zip(
-                a.args[-len(a.defaults) :],
+                a.args[-len(a.defaults):],
                 [serialize_data(d) for d in a.defaults],
             )
         )
         adefaults["kwargs"] = dict(
-            zip(a.args[-len(a.defaults) :], [d for d in a.defaults])
+            zip(a.args[-len(a.defaults):], [d for d in a.defaults])
         )
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: myxie
body: I actually think this is a little more complicated. 
created_at: 2025-04-09T02:36:05Z
author_association: COLLABORATOR
diff_hunk: @@ -75,12 +75,12 @@
     if a.defaults:
         fdefaults["kwargs"] = dict(
             zip(
-                a.args[-len(a.defaults) :],
+                a.args[-len(a.defaults):],
                 [serialize_data(d) for d in a.defaults],
             )
         )
         adefaults["kwargs"] = dict(
-            zip(a.args[-len(a.defaults) :], [d for d in a.defaults])
+            zip(a.args[-len(a.defaults):], [d for d in a.defaults])
         )
path: daliuge-engine/dlg/apps/pyfunc.py
user: login: sourcery-ai[bot]
body: I understand your concern, @myxie. While the Sourcery suggestion looks straightforward, there might be nuanced reasons why the current implementation exists. 

Could you elaborate on why you think this is more complicated? Are there specific edge cases or behaviors in the function signature handling that make a direct conversion problematic? The context of handling function defaults and serialization suggests there might be subtle requirements that aren't immediately apparent.

I'm happy to discuss the potential implications of changing this code and understand the reasoning behind the current implementation. If you can provide more context about the potential complexities, we can collaboratively determine the best approach.
created_at: 2025-04-09T02:36:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,145 @@
+#
+#    ICRAR - International Centre for Radio Astronomy Research
+#    (c) UWA - The University of Western Australia, 2014
+#    Copyright by UWA (in the framework of the ICRAR)
+#    All rights reserved
+#
+#    This library is free software; you can redistribute it and/or
+#    modify it under the terms of the GNU Lesser General Public
+#    License as published by the Free Software Foundation; either
+#    version 2.1 of the License, or (at your option) any later version.
+#
+#    This library is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+#    Lesser General Public License for more details.
+#
+#    You should have received a copy of the GNU Lesser General Public
+#    License along with this library; if not, write to the Free Software
+#    Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+#    MA 02111-1307  USA
+#
+
+"""
+Module with utility methods to create runtime file paths for the DALiuGE Engine.
+
+The Engine supports implicit filepath naming if FileDROPs have no user-defined name in
+the Logical Graph Template. This helps when multiple FileDROPs are created in a
+construct (e.g. Scatter or Loop) and we don't run the risk of overwriting them.
+
+This module provides the methods that support different filepath naming heuristics to
+facilitate implicit filepath naming at runtime.
+"""
+import datetime
+import logging
+import re
+import os
+import uuid
+
+NON_FILENAME_CHARACTERS = re.compile(r":|%s" % os.sep)
+
+def default_map():
+    """
+    Get the default map for the FSTRING replacement keywords
+    """
+    return {
+        "dlg": "DALIGUE",
+        "datetime": datetime.date.today().strftime("%Y-%m-%d"),
+        "uid": str(uuid.uuid4()),
+    } 
+
+def base_uid_filename(uid: str, humanKey: str):
+    """
+    This a basic filename generator, using the UID and humandReadableKey. The function
+    returns only the name of the file, and expects the full filepath to be handled by
+    the function caller.
+
+    Parameters
+    ----------
+    uid: str, the UID of the DROP
+    humanKey: str, a human-readable key (typically a truncated UID) for user-facing
+        display
+
+    Notes
+    -----
+    An example DROP UID would be:
+
+        '2024-10-30T12:01:57_0140555b-8c23-4d6a-9e24-e16c15555e8c_0'
+
+    This approach will lead to an abridged UID along with the humanKey appended:
+
+       '2024-10-30T12:01:57_0140555b_0/1/1'
+
+    In this case, 0/1/1 is the "humanKey", derived from the DALiuGE Translator
+    constructs (For more information on humanKey generation, refer to
+    dlg.common.truncateUidToKey).
+
+    The final part of the method replaces characters that cannot be in filenames from
+    the UIDs (e.g. "/").
+
+    Returns
+    -------
+        str: Base filename
+    """
+    if not uid or not humanKey:
+        return None
+    fn = uid.split("_")[0] + "_" + str(humanKey)
path: daliuge-engine/dlg/data/path_builder.py
user: login: myxie
body: I can't guarantee the input here so this is preventative. 
created_at: 2025-04-09T03:04:04Z
author_association: COLLABORATOR
diff_hunk: @@ -84,17 +117,29 @@ def serialize_applicationArgs(applicationArgs, prefix="--", separator=" "):
     logger.info("Constructed command line arguments: %s %s", pargs, kwargs)
     return (pargs, skwargs)
 
+def extract_encoded_value(args: dict, drop: "AbstractDROP", encoding:DropParser):
+    """
+
+    Parameters
+    ----------
+    args
+    drop
+    encoding
+
+    Returns
+    -------
+
+    """
 
 def identify_named_ports(
path: daliuge-engine/dlg/named_port_utils.py
user: login: myxie
body: `named_port_utils` has a lot of ugly code - it will continue to be a work in progress. 
created_at: 2025-04-09T03:05:13Z
author_association: COLLABORATOR

PR comments:
user: login: sourcery-ai[bot]
created_at: 2025-03-12T11:50:38Z
body: <!-- Generated by sourcery-ai[bot]: start review_guide -->

## Reviewer's Guide by Sourcery

This pull request introduces significant enhancements to the DALiuGE engine, specifically focusing on improving FileDROP naming and providing better support for input-output filepaths. The changes involve modifications to the PyFuncApp class, allowing for more control over side-effect files, along with updates to unit tests, build workflows, and the translator to accommodate these new features.

_No diagrams generated as the changes look simple and do not need a visual representation._

### File-Level Changes

| Change | Details | Files |
| ------ | ------- | ----- |
| Modified the `PyFuncApp` class to allow mapping of InputOutput/OutputPorts to function parameters, enabling control over side-effect files. | <ul><li>Added support for multiple InputPorts to DataDROPs.</li><li>Added support to map InputOutput/OutputPorts to function parameters of a PyFuncApp drop.</li><li>Restructured pyfunc.py to improve readability and isolate specific functionality.</li><li>Added a `_populate_arguments_with_graph_data` method to populate application arguments with data from the logical graph.</li><li>Added a `_map_parameters_to_func_args` method to map EAGLE/DALiuGE input parameters to function arguments.</li><li>Added a `_update_filepaths` method to map InputOutput attributes to filepaths.</li><li>Modified the `_ports2args` method to replace arguments with values from ports.</li><li>Added a dataclass `Argument` to encapsulate data associated with all arguments.</li><li>Added an Enum `ArgType` to represent positional or keyword arguments.</li><li>Added a `_initialise_args` method to ensure default arguments are properly setup.</li><li>Added a `_get_arg_info` method to get necessary information for populating the argument.</li><li>Added a `populate_arguments_without_signature` flag to handle functions without signatures.</li><li>Added a `_clean_applicationArgs` method to remove function definition arguments in applicationArgs.</li><li>Added a `_match_parser` method to match the output parser to the appropriate drop.</li><li>Added a `_apply_filename_modifiers` method to apply prefix and suffix modifiers to the filename.</li></ul> | `daliuge-engine/dlg/apps/pyfunc.py`<br/>`daliuge-engine/dlg/named_port_utils.py`<br/>`daliuge-engine/dlg/data/drops/file.py`<br/>`daliuge-engine/dlg/data/drops/data_base.py`<br/>`daliuge-engine/dlg/apps/simple_functions.py`<br/>`daliuge-engine/dlg/data/path_builder.py` |
| Added new unit tests and updated existing tests to validate the changes related to FileDROPs and named ports. | <ul><li>Created new test files for FileDROPs and path builders.</li><li>Updated existing test files for named port apps.</li><li>Added tests for string2json functions.</li><li>Added tests for filepaths with multiple apps.</li><li>Added tests for filepaths with memory drop file input.</li></ul> | `daliuge-engine/test/test_named_port_apps.py`<br/>`daliuge-engine/test/named_ports/test_filedrops.py`<br/>`daliuge-engine/test/named_ports/test_path_builder.py`<br/>`daliuge-engine/test/dlg_end_to_end_utils.py`<br/>`daliuge-engine/test/apps/test_pyfunc.py` |
| Updated the build and documentation workflows. | <ul><li>Updated the ubuntu version from 20.04 to 22.04 in the github workflows.</li><li>Updated the dependency from merklelib >= 1.0 to np-merklelib >= 1.0 in docs/requirements.txt.</li><li>Updated the dependency from dlg_paletteGen to dlg-paletteGen in .github/workflows/create-palettes.yml.</li></ul> | `.github/workflows/create-palettes.yml`<br/>`docs/requirements.txt`<br/>`.github/workflows/build-documentation.yml`<br/>`.github/workflows/linting.yml`<br/>`.github/workflows/run-unit-tests.yml` |
| Modified the translator to link drops and handle port mapping. | <ul><li>Modified the `_link_drops` method to handle port mapping.</li><li>Modified the `_getPortName` method to handle port names.</li><li>Modified the `unroll_to_tpl` method to handle group data drops.</li></ul> | `daliuge-translator/dlg/dropmake/lg.py`<br/>`daliuge-translator/dlg/dropmake/lg_node.py` |
| Updated the common drop dictionary. | <ul><li>Added supported keys for common, AppDROP, DataDROP and FileDROP.</li></ul> | `daliuge-common/dlg/common/__init__.py` |

---

<details>
<summary>Tips and commands</summary>

#### Interacting with Sourcery

- **Trigger a new review:** Comment `@sourcery-ai review` on the pull request.
- **Continue discussions:** Reply directly to Sourcery's review comments.
- **Generate a GitHub issue from a review comment:** Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with `@sourcery-ai issue` to create an issue from it.
- **Generate a pull request title:** Write `@sourcery-ai` anywhere in the pull
  request title to generate a title at any time. You can also comment
  `@sourcery-ai title` on the pull request to (re-)generate the title at any time.
- **Generate a pull request summary:** Write `@sourcery-ai summary` anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment `@sourcery-ai summary` on the pull request to
  (re-)generate the summary at any time.
- **Generate reviewer's guide:** Comment `@sourcery-ai guide` on the pull
  request to (re-)generate the reviewer's guide at any time.
- **Resolve all Sourcery comments:** Comment `@sourcery-ai resolve` on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
- **Dismiss all Sourcery reviews:** Comment `@sourcery-ai dismiss` on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  `@sourcery-ai review` to trigger a new review!
- **Generate a plan of action for an issue:** Comment `@sourcery-ai plan` on
  an issue to generate a plan of action for it.

#### Customizing Your Experience

Access your [dashboard](https://app.sourcery.ai) to:
- Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
- Change the review language.
- Add, remove or edit custom review instructions.
- Adjust other review settings.

#### Getting Help

- [Contact our support team](mailto:support@sourcery.ai) for questions or feedback.
- Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information.
- Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).

</details>

<!-- Generated by sourcery-ai[bot]: end review_guide -->
author_association: CONTRIBUTOR
user: login: coveralls
created_at: 2025-04-09T02:06:49Z
body: 
[![Coverage Status](https://coveralls.io/builds/73362594/badge)](https://coveralls.io/builds/73362594)

coverage: 78.955% (+0.3%) from 78.626%
when pulling **c7c691c05a3411568abd2e9dcd24704a1642ff4a on LIU-5**
into **3224f45f4d41bc29d55495119f2fae17a9ce7fbc on master**.

author_association: NONE
user: login: myxie
created_at: 2025-04-09T03:49:45Z
body: @awicenec may you please provide a review of this PR? It would be great to (finally) get these changes in! 
author_association: COLLABORATOR
user: login: awicenec
created_at: 2025-04-23T11:43:05Z
body: I have now updated the [side-effect graph]( http://localhost:8888/?service=GitHub&repository=ICRAR/EAGLE_test_repo&branch=master&path=eagle_test_graphs/daliuge_tests/engine/test_filedrops&filename=Filepath_test_side_effect.graph) to include everything I think should work. This now captures the actual return value of the function (as it would for a normal function) as well as the side-effect written by the function internally and also a follow on copy of the hello.txt file. That begs the question, why we would need the double connection at all? The hello message is written internally by the function to the file with the name provided through the filepath argument and if the function would not return anything, the second file would not contain anything. If it would work that way, that would be very clean. Right now, when this graph is executed there are four files generated:

1. The hello.txt file with the correct message,
2. The output.txt file with the return value as a string (this is how it is defined in the graph),
3. The hello2.txt file with the _**return value**_ in pickled format (this is clearly wrong!),
4. An additional file with a DALiuGE generated filename with the return value pickled. (this should not exist).

author_association: CONTRIBUTOR
user: login: myxie
created_at: 2025-04-24T08:33:35Z
body: @awicenec  I have used the side-effect graph (with some updates) and it is currently sitting in the LIU-5 branch of the eagle-test-graphs: https://github.com/ICRAR/EAGLE_test_repo/pull/17. 
author_association: COLLABORATOR