Repository: dask/dask-jobqueue
Pull request #577 (2022-08-19 10:46:51)

rename `job_extra`
to `job_extra_directives`, see #323.

If `job_extra_directives` is empty,  `job_extra` will still be respected.

The warning code is duplicated for different *Cluster classes now. I thought about putting it in the base class, but that does not know this parameter (almost all implentations define it, but not all) and it is not treated exactly the same everywhere.
So I think the repetition is ok for this temporary code.

guillaumeeb: Thanks @jolange for continuing work here, I really appreciate it.

> The warning code is duplicated for different *Cluster classes now. I thought about putting it in the base class, but that does not know this parameter (almost all implentations define it, but not all) and it is not treated exactly the same everywhere

With a quick glance, I'm not sure why it is duplicated and not in the base class. Do you think something prevent to declare a private attribute in the main class, putting the warning in there, and just keep special handling in defferent implementations?
jolange: I did not check in detail before, which *Cluster classes do not implement it, but it now I see that it's actually only Moab and that this basically is PBS. So all of them use the parameter, really.

The only problem is that the parameter is supposed to be a dict for HTCondor, but a list for every other one.
guillaumeeb: > The only problem is that the parameter is supposed to be a dict for HTCondor, but a list for every other one.

I guess this is a problem only for the doc string or warning message? Maybe we could indicate list or dict as type? 

I'm not sure if we should duplicate code just for this? Or maybe we should just do something for the HTCondor specific case?
jolange: Only for the doc string, actually. Indicating "dict or list" would be ok, I think, but the doc strings for the *Cluster classes are rather different (and rightfully so, I think):
```console
$ ack "job_extra_directives : " dask_jobqueue/*.py -A2
dask_jobqueue/htcondor.py
245:    job_extra_directives : dict
246-        Extra submit file attributes for the job as key-value pairs.
247-        They will be inserted as ``key = value``.

dask_jobqueue/lsf.py
211:    job_extra_directives : list
212-        List of other LSF options, for example -u. Each option will be
213-        prepended with the #LSF prefix.

dask_jobqueue/oar.py
134:    job_extra_directives : list
135-        List of other OAR options, for example `-t besteffort`. Each option will be prepended with the #OAR prefix.
136-

dask_jobqueue/pbs.py
143:    job_extra_directives : list
144-        List of other PBS options. Each option will be prepended with the #PBS prefix.
145-

dask_jobqueue/sge.py
117:    job_extra_directives : list
118-        List of other SGE options, for example -w e. Each option will be
119-        prepended with the #$ prefix.

dask_jobqueue/slurm.py
148:    job_extra_directives : list
149-        List of other Slurm options, for example -j oe. Each option will be prepended with the #SBATCH prefix.
150-

```

What do you think about making it a parameter of the base class and store it as member variable, but keeping the doc string in the different base classes? Maybe adding a comment to the parameter in the base class where the doc string(s) can be found?
guillaumeeb: Sounds good. We should put a generic doc string in the base class, and some more complete description in each cluster implementation.
guillaumeeb: Thanks a lot @jolange for sticking with that!!

Patch:
diff --git a/dask_jobqueue/core.py b/dask_jobqueue/core.py
index 09dc5c9..4f97c07 100644
--- a/dask_jobqueue/core.py
+++ b/dask_jobqueue/core.py
@@ -107,6 +107,11 @@ class Job(ProcessInterface, abc.ABC):
     Parameters
     ----------
     {job_parameters}
+    job_extra : list or dict
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : list or dict
+        Unused in this base class:
+        List or dict of other options for the queueing system. See derived classes for specific descriptions.
 
     Attributes
     ----------
@@ -159,6 +164,8 @@ class Job(ProcessInterface, abc.ABC):
         local_directory=None,
         extra=None,
         worker_extra_args=None,
+        job_extra=None,
+        job_extra_directives=None,
         env_extra=None,
         job_script_prologue=None,
         header_skip=None,
@@ -226,6 +233,26 @@ class Job(ProcessInterface, abc.ABC):
             warnings.warn(warn, FutureWarning)
             if not worker_extra_args:
                 worker_extra_args = extra
+
+        if job_extra is None:
+            job_extra = dask.config.get("jobqueue.%s.job-extra" % self.config_name, [])
+        if job_extra_directives is None:
+            job_extra_directives = dask.config.get(
+                "jobqueue.%s.job-extra-directives" % self.config_name, []
+            )
+        if job_extra is not None:
+            warn = (
+                "job_extra has been renamed to job_extra_directives. "
+                "You are still using it (even if only set to []; please also check config files). "
+                "If you did not set job_extra_directives yet, job_extra will be respected for now, "
+                "but it will be removed in a future release. "
+                "If you already set job_extra_directives, job_extra is ignored and you can remove it."
+            )
+            warnings.warn(warn, FutureWarning)
+            if not job_extra_directives:
+                job_extra_directives = job_extra
+        self.job_extra_directives = job_extra_directives
+
         if env_extra is None:
             env_extra = dask.config.get("jobqueue.%s.env-extra" % self.config_name)
         if job_script_prologue is None:
diff --git a/dask_jobqueue/htcondor.py b/dask_jobqueue/htcondor.py
index f3d5a96..b7a7b23 100644
--- a/dask_jobqueue/htcondor.py
+++ b/dask_jobqueue/htcondor.py
@@ -37,7 +37,6 @@ Queue
         scheduler=None,
         name=None,
         disk=None,
-        job_extra=None,
         config_name=None,
         submit_command_extra=None,
         cancel_command_extra=None,
@@ -54,12 +53,6 @@ Queue
                 "You must specify how much disk to use per job like ``disk='1 GB'``"
             )
         self.worker_disk = parse_bytes(disk)
-        if job_extra is None:
-            self.job_extra = dask.config.get(
-                "jobqueue.%s.job-extra" % self.config_name, {}
-            )
-        else:
-            self.job_extra = job_extra
 
         if self._job_script_prologue is not None:
             # Overwrite command template: prepend commands from job_script_prologue separated by semicolon.
@@ -94,8 +87,8 @@ Queue
                     "Stream_Error": True,
                 }
             )
-        if self.job_extra:
-            self.job_header_dict.update(self.job_extra)
+        if self.job_extra_directives:
+            self.job_header_dict.update(self.job_extra_directives)
 
         if submit_command_extra is None:
             submit_command_extra = dask.config.get(
@@ -225,7 +218,10 @@ class HTCondorCluster(JobQueueCluster):
     disk : str
         Total amount of disk per job
     job_extra : dict
-        Extra submit file attributes for the job
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : dict
+        Extra submit file attributes for the job as key-value pairs.
+        They will be inserted as ``key = value``.
     submit_command_extra : list of str
         Extra arguments to pass to condor_submit
     cancel_command_extra : list of str
diff --git a/dask_jobqueue/jobqueue.yaml b/dask_jobqueue/jobqueue.yaml
index 61f2ecc..5deae27 100644
--- a/dask_jobqueue/jobqueue.yaml
+++ b/dask_jobqueue/jobqueue.yaml
@@ -22,7 +22,8 @@ jobqueue:
     env-extra: null
     job-script-prologue: []
     resource-spec: null
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     
     # Scheduler options
@@ -51,7 +52,8 @@ jobqueue:
     env-extra: null
     job-script-prologue: []
     resource-spec: null
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     
     # Scheduler options
@@ -79,7 +81,8 @@ jobqueue:
     walltime: '00:30:00'
     env-extra: null
     job-script-prologue: []
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     resource-spec: null
     
@@ -110,7 +113,8 @@ jobqueue:
     job-script-prologue: []
     job-cpu: null
     job-mem: null
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     
     # Scheduler options
@@ -139,7 +143,8 @@ jobqueue:
     env-extra: null
     job-script-prologue: []
     resource-spec: null
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     
     # Scheduler options
@@ -169,7 +174,8 @@ jobqueue:
     job-script-prologue: []
     ncpus: null
     mem: null
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     lsf-units: null
     use-stdin: True             # (bool) How jobs are launched, i.e. 'bsub jobscript.sh' or 'bsub < jobscript.sh'
@@ -196,7 +202,8 @@ jobqueue:
     disk: null                  # Total amount of disk per job
     env-extra: null
     job-script-prologue: []
-    job-extra: {}               # Extra submit attributes
+    job-extra: null             # Extra submit attributes
+    job-extra-directives: {}    # Extra submit attributes
     submit-command-extra: []    # Extra condor_submit arguments
     cancel-command-extra: []    # Extra condor_rm arguments
     log-directory: null
@@ -221,7 +228,8 @@ jobqueue:
 
     env-extra: null
     job-script-prologue: []
-    job-extra: []
+    job-extra: null
+    job-extra-directives: []
     log-directory: null
     
     # Scheduler options
diff --git a/dask_jobqueue/local.py b/dask_jobqueue/local.py
index cd87ce5..e7b0aa6 100644
--- a/dask_jobqueue/local.py
+++ b/dask_jobqueue/local.py
@@ -32,7 +32,6 @@ class LocalJob(Job):
         project=None,
         resource_spec=None,
         walltime=None,
-        job_extra=None,
         config_name=None,
         **kwargs
     ):
diff --git a/dask_jobqueue/lsf.py b/dask_jobqueue/lsf.py
index 3675976..0d6c7db 100644
--- a/dask_jobqueue/lsf.py
+++ b/dask_jobqueue/lsf.py
@@ -28,7 +28,6 @@ class LSFJob(Job):
         ncpus=None,
         mem=None,
         walltime=None,
-        job_extra=None,
         lsf_units=None,
         config_name=None,
         use_stdin=None,
@@ -48,8 +47,6 @@ class LSFJob(Job):
             mem = dask.config.get("jobqueue.%s.mem" % self.config_name)
         if walltime is None:
             walltime = dask.config.get("jobqueue.%s.walltime" % self.config_name)
-        if job_extra is None:
-            job_extra = dask.config.get("jobqueue.%s.job-extra" % self.config_name)
         if lsf_units is None:
             lsf_units = dask.config.get("jobqueue.%s.lsf-units" % self.config_name)
 
@@ -96,7 +93,7 @@ class LSFJob(Job):
             header_lines.append("#BSUB -M %s" % memory_string)
         if walltime is not None:
             header_lines.append("#BSUB -W %s" % walltime)
-        header_lines.extend(["#BSUB %s" % arg for arg in job_extra])
+        header_lines.extend(["#BSUB %s" % arg for arg in self.job_extra_directives])
 
         # Declare class attribute that shall be overridden
         self.job_header = "\n".join(header_lines)
@@ -189,6 +186,8 @@ class LSFCluster(JobQueueCluster):
         Walltime for each worker job in HH:MM. Passed to `#BSUB -W` option.
     {cluster}
     job_extra : list
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : list
         List of other LSF options, for example -u. Each option will be
         prepended with the #LSF prefix.
     lsf_units : str
diff --git a/dask_jobqueue/oar.py b/dask_jobqueue/oar.py
index 13e8cce..0e8c0d8 100644
--- a/dask_jobqueue/oar.py
+++ b/dask_jobqueue/oar.py
@@ -24,7 +24,6 @@ class OARJob(Job):
         project=None,
         resource_spec=None,
         walltime=None,
-        job_extra=None,
         config_name=None,
         **base_class_kwargs
     ):
@@ -42,8 +41,6 @@ class OARJob(Job):
             )
         if walltime is None:
             walltime = dask.config.get("jobqueue.%s.walltime" % self.config_name)
-        if job_extra is None:
-            job_extra = dask.config.get("jobqueue.%s.job-extra" % self.config_name)
 
         header_lines = []
         if self.job_name is not None:
@@ -67,7 +64,7 @@ class OARJob(Job):
 
         full_resource_spec = ",".join(resource_spec_list)
         header_lines.append("#OAR -l %s" % full_resource_spec)
-        header_lines.extend(["#OAR %s" % arg for arg in job_extra])
+        header_lines.extend(["#OAR %s" % arg for arg in self.job_extra_directives])
 
         self.job_header = "\n".join(header_lines)
 
@@ -113,6 +110,8 @@ class OARCluster(JobQueueCluster):
     walltime : str
         Walltime for each worker job.
     job_extra : list
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : list
         List of other OAR options, for example `-t besteffort`. Each option will be prepended with the #OAR prefix.
 
     Examples
diff --git a/dask_jobqueue/pbs.py b/dask_jobqueue/pbs.py
index 438858e..289d9e9 100644
--- a/dask_jobqueue/pbs.py
+++ b/dask_jobqueue/pbs.py
@@ -47,7 +47,6 @@ class PBSJob(Job):
         project=None,
         resource_spec=None,
         walltime=None,
-        job_extra=None,
         config_name=None,
         **base_class_kwargs
     ):
@@ -63,8 +62,7 @@ class PBSJob(Job):
             )
         if walltime is None:
             walltime = dask.config.get("jobqueue.%s.walltime" % self.config_name)
-        if job_extra is None:
-            job_extra = dask.config.get("jobqueue.%s.job-extra" % self.config_name)
+
         if project is None:
             project = dask.config.get(
                 "jobqueue.%s.project" % self.config_name
@@ -97,7 +95,7 @@ class PBSJob(Job):
         if self.log_directory is not None:
             header_lines.append("#PBS -e %s/" % self.log_directory)
             header_lines.append("#PBS -o %s/" % self.log_directory)
-        header_lines.extend(["#PBS %s" % arg for arg in job_extra])
+        header_lines.extend(["#PBS %s" % arg for arg in self.job_extra_directives])
 
         # Declare class attribute that shall be overridden
         self.job_header = "\n".join(header_lines)
@@ -121,6 +119,8 @@ class PBSCluster(JobQueueCluster):
     walltime : str
         Walltime for each worker job.
     job_extra : list
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : list
         List of other PBS options. Each option will be prepended with the #PBS prefix.
 
     Examples
diff --git a/dask_jobqueue/sge.py b/dask_jobqueue/sge.py
index 10fa24a..9cb113c 100644
--- a/dask_jobqueue/sge.py
+++ b/dask_jobqueue/sge.py
@@ -20,7 +20,6 @@ class SGEJob(Job):
         project=None,
         resource_spec=None,
         walltime=None,
-        job_extra=None,
         config_name=None,
         **base_class_kwargs
     ):
@@ -38,8 +37,6 @@ class SGEJob(Job):
             )
         if walltime is None:
             walltime = dask.config.get("jobqueue.%s.walltime" % self.config_name)
-        if job_extra is None:
-            job_extra = dask.config.get("jobqueue.%s.job-extra" % self.config_name)
 
         header_lines = []
         if self.job_name is not None:
@@ -56,7 +53,7 @@ class SGEJob(Job):
             header_lines.append("#$ -e %(log_directory)s/")
             header_lines.append("#$ -o %(log_directory)s/")
         header_lines.extend(["#$ -cwd", "#$ -j y"])
-        header_lines.extend(["#$ %s" % arg for arg in job_extra])
+        header_lines.extend(["#$ %s" % arg for arg in self.job_extra_directives])
         header_template = "\n".join(header_lines)
 
         config = {
@@ -96,6 +93,8 @@ class SGECluster(JobQueueCluster):
     walltime : str
         Walltime for each worker job.
     job_extra : list
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : list
         List of other SGE options, for example -w e. Each option will be
         prepended with the #$ prefix.
 
diff --git a/dask_jobqueue/slurm.py b/dask_jobqueue/slurm.py
index a46662e..4cea7a5 100644
--- a/dask_jobqueue/slurm.py
+++ b/dask_jobqueue/slurm.py
@@ -23,7 +23,6 @@ class SLURMJob(Job):
         walltime=None,
         job_cpu=None,
         job_mem=None,
-        job_extra=None,
         config_name=None,
         **base_class_kwargs
     ):
@@ -41,8 +40,6 @@ class SLURMJob(Job):
             job_cpu = dask.config.get("jobqueue.%s.job-cpu" % self.config_name)
         if job_mem is None:
             job_mem = dask.config.get("jobqueue.%s.job-mem" % self.config_name)
-        if job_extra is None:
-            job_extra = dask.config.get("jobqueue.%s.job-extra" % self.config_name)
 
         header_lines = []
         # SLURM header build
@@ -77,7 +74,7 @@ class SLURMJob(Job):
 
         if walltime is not None:
             header_lines.append("#SBATCH -t %s" % walltime)
-        header_lines.extend(["#SBATCH %s" % arg for arg in job_extra])
+        header_lines.extend(["#SBATCH %s" % arg for arg in self.job_extra_directives])
 
         # Declare class attribute that shall be overridden
         self.job_header = "\n".join(header_lines)
@@ -127,6 +124,8 @@ class SLURMCluster(JobQueueCluster):
         Amount of memory to request in SLURM. If None, defaults to worker
         processes * memory
     job_extra : list
+        Deprecated: use ``job_extra_directives`` instead. This parameter will be removed in a future version.
+    job_extra_directives : list
         List of other Slurm options, for example -j oe. Each option will be prepended with the #SBATCH prefix.
 
     Examples
diff --git a/docs/source/debug.rst b/docs/source/debug.rst
index b48056c..2781a5f 100644
--- a/docs/source/debug.rst
+++ b/docs/source/debug.rst
@@ -27,7 +27,8 @@ a real job script file, and submit it using ``qsub``, ``sbatch``, ``bsub`` or
 what is appropriate for you job queuing system.
 
 To correct any problem detected at this point, you could try to use
-``job_extra`` or ``job_script_prologue`` kwargs when initializing your cluster object.
+``job_extra_directives`` or ``job_script_prologue`` kwargs when initializing
+your cluster object.
 
 In particular, pay attention to the python executable used to launch the
 workers, which by default is the one used to launch the scheduler (this makes
diff --git a/docs/source/examples.rst b/docs/source/examples.rst
index d52dbd8..118c57b 100644
--- a/docs/source/examples.rst
+++ b/docs/source/examples.rst
@@ -42,13 +42,15 @@ can be used, called ``MoabCluster``:
    import os
    from dask_jobqueue import MoabCluster
 
-   cluster = MoabCluster(cores=6,
-                         processes=6,
-                         project='gfdl_m',
-                         memory='16G',
-                         resource_spec='pmem=96G',
-                         job_extra=['-d /home/First.Last', '-M none'],
-                         local_directory=os.getenv('TMPDIR', '/tmp'))
+   cluster = MoabCluster(
+       cores=6,
+       processes=6,
+       project="gfdl_m",
+       memory="16G",
+       resource_spec="pmem=96G",
+       job_extra_directives=["-d /home/First.Last", "-M none"],
+       local_directory=os.getenv("TMPDIR", "/tmp"),
+   )
 
 SGE Deployments
 ---------------
@@ -124,17 +126,19 @@ SLURM Deployment: Low-priority node usage
 
     from dask_jobqueue import SLURMCluster
 
-    cluster = SLURMCluster(cores=24,
-                           processes=6,
-                           memory="16GB",
-                           project="co_laika",
-                           queue='savio2_bigmem',
-                           job_script_prologue=[
-                               'export LANG="en_US.utf8"',
-                               'export LANGUAGE="en_US.utf8"',
-                               'export LC_ALL="en_US.utf8"'
-                           ],
-                           job_extra=['--qos="savio_lowprio"'])
+    cluster = SLURMCluster(
+        cores=24,
+        processes=6,
+        memory="16GB",
+        project="co_laika",
+        queue="savio2_bigmem",
+        job_script_prologue=[
+            'export LANG="en_US.utf8"',
+            'export LANGUAGE="en_US.utf8"',
+            'export LC_ALL="en_US.utf8"',
+        ],
+        job_extra_directives=['--qos="savio_lowprio"'],
+    )
 
 
 


Test patch:
diff --git a/dask_jobqueue/tests/test_htcondor.py b/dask_jobqueue/tests/test_htcondor.py
index d1228c0..c5c5bf6 100644
--- a/dask_jobqueue/tests/test_htcondor.py
+++ b/dask_jobqueue/tests/test_htcondor.py
@@ -33,7 +33,7 @@ def test_job_script():
             "cd /some/path/",
             "source venv/bin/activate",
         ],
-        job_extra={"+Extra": "True"},
+        job_extra_directives={"+Extra": "True"},
         submit_command_extra=["-verbose"],
         cancel_command_extra=["-forcex"],
     ) as cluster:
@@ -134,7 +134,8 @@ def test_config_name_htcondor_takes_custom_config():
         "cores": 1,
         "memory": "120 MB",
         "disk": "120 MB",
-        "job-extra": [],
+        "job-extra": None,
+        "job-extra-directives": [],
         "name": "myname",
         "processes": 1,
         "interface": None,
diff --git a/dask_jobqueue/tests/test_job.py b/dask_jobqueue/tests/test_job.py
index 1ada7b6..d531317 100644
--- a/dask_jobqueue/tests/test_job.py
+++ b/dask_jobqueue/tests/test_job.py
@@ -271,3 +271,70 @@ def test_deprecation_extra(Cluster):
     )
     job_script = job.job_script()
     assert "old_param" in job_script
+
+
+def test_deprecation_job_extra(Cluster):
+    if issubclass(Cluster, LocalCluster):
+        return  # nothing to test in this implentation, really
+
+    import warnings
+
+    # test issuing of warning
+    warnings.simplefilter("always")
+
+    job_cls = Cluster.job_cls
+    with warnings.catch_warnings(record=True) as w:
+        # should give a warning
+        job = job_cls(cores=1, memory="1 GB", job_extra={"old_param": "1"})
+        assert len(w) == 1
+        assert issubclass(w[0].category, FutureWarning)
+        assert "job_extra has been renamed" in str(w[0].message)
+    with warnings.catch_warnings(record=True) as w:
+        # should give a warning
+        job = job_cls(
+            cores=1,
+            memory="1 GB",
+            job_extra={"old_param": "1"},
+            job_extra_directives={"new_param": "2"},
+        )
+        assert len(w) == 1
+        assert issubclass(w[0].category, FutureWarning)
+        assert "job_extra has been renamed" in str(w[0].message)
+    with warnings.catch_warnings(record=True) as w:
+        # should not give a warning
+        job = job_cls(
+            cores=1,
+            memory="1 GB",
+            job_extra_directives={"new_param": "2"},
+        )
+        assert len(w) == 0
+
+    # the rest is not about the warning but about behaviour: if job_extra_directives is not
+    # set, job_extra should still be used if provided
+    warnings.simplefilter("ignore")
+    job = job_cls(
+        cores=1,
+        memory="1 GB",
+        job_extra={"old_param": "1"},
+        job_extra_directives={"new_param": "2"},
+    )
+    job_script = job.job_script()
+    assert "old_param" not in job_script
+    assert "new_param" in job_script
+
+    job = job_cls(
+        cores=1,
+        memory="1 GB",
+        job_extra={"old_param": "1"},
+    )
+    job_script = job.job_script()
+    assert "old_param" in job_script
+
+    job = job_cls(
+        cores=1,
+        memory="1 GB",
+        job_extra={"old_param": "1"},
+        job_extra_directives=[],
+    )
+    job_script = job.job_script()
+    assert "old_param" in job_script
diff --git a/dask_jobqueue/tests/test_lsf.py b/dask_jobqueue/tests/test_lsf.py
index 759de48..40c826c 100644
--- a/dask_jobqueue/tests/test_lsf.py
+++ b/dask_jobqueue/tests/test_lsf.py
@@ -73,7 +73,7 @@ def test_header():
         assert "#BSUB -P" not in cluster.job_header
 
     with LSFCluster(
-        cores=4, memory="8GB", job_extra=["-u email@domain.com"]
+        cores=4, memory="8GB", job_extra_directives=["-u email@domain.com"]
     ) as cluster:
 
         assert "#BSUB -u email@domain.com" in cluster.job_header
@@ -139,7 +139,7 @@ def test_job_script():
         cores=1,
         memory="16GB",
         project="Dask On LSF",
-        job_extra=["-R rusage[mem=16GB]"],
+        job_extra_directives=["-R rusage[mem=16GB]"],
     ) as cluster:
 
         job_script = cluster.job_script()
@@ -312,7 +312,8 @@ def test_config_name_lsf_takes_custom_config():
         "mem": 2,
         "memory": "2 GB",
         "walltime": "00:02",
-        "job-extra": [],
+        "job-extra": None,
+        "job-extra-directives": [],
         "lsf-units": "TB",
         "name": "myname",
         "processes": 1,
diff --git a/dask_jobqueue/tests/test_oar.py b/dask_jobqueue/tests/test_oar.py
index 3c802ac..20a4b8e 100644
--- a/dask_jobqueue/tests/test_oar.py
+++ b/dask_jobqueue/tests/test_oar.py
@@ -21,7 +21,7 @@ def test_header():
         processes=4,
         cores=8,
         memory="28GB",
-        job_extra=["-t besteffort"],
+        job_extra_directives=["-t besteffort"],
     ) as cluster:
         assert "walltime=" in cluster.job_header
         assert "#OAR --project DaskOnOar" in cluster.job_header
@@ -101,7 +101,8 @@ def test_config_name_oar_takes_custom_config():
         "cores": 1,
         "memory": "2 GB",
         "walltime": "00:02",
-        "job-extra": [],
+        "job-extra": None,
+        "job-extra-directives": [],
         "name": "myname",
         "processes": 1,
         "interface": None,
diff --git a/dask_jobqueue/tests/test_pbs.py b/dask_jobqueue/tests/test_pbs.py
index 69027d3..11f1c04 100644
--- a/dask_jobqueue/tests/test_pbs.py
+++ b/dask_jobqueue/tests/test_pbs.py
@@ -50,7 +50,7 @@ def test_header(Cluster):
         assert "#PBS -A" not in cluster.job_header
         assert "#PBS -q" not in cluster.job_header
 
-    with Cluster(cores=4, memory="8GB", job_extra=["-j oe"]) as cluster:
+    with Cluster(cores=4, memory="8GB", job_extra_directives=["-j oe"]) as cluster:
 
         assert "#PBS -j oe" in cluster.job_header
         assert "#PBS -N" in cluster.job_header
@@ -115,7 +115,7 @@ def test_basic(loop):
         cores=2,
         memory="2GiB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
         with Client(cluster) as client:
@@ -150,7 +150,7 @@ def test_scale_cores_memory(loop):
         cores=2,
         memory="2GiB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
         with Client(cluster) as client:
@@ -185,7 +185,7 @@ def test_basic_scale_edge_cases(loop):
         cores=2,
         memory="2GB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
 
@@ -210,7 +210,7 @@ def test_adaptive(loop):
         cores=2,
         memory="2GB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
         cluster.adapt()
@@ -240,7 +240,7 @@ def test_adaptive_grouped(loop):
         cores=2,
         memory="2GB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
         cluster.adapt(minimum=1)  # at least 1 worker
@@ -265,7 +265,7 @@ def test_adaptive_cores_mem(loop):
         cores=2,
         memory="2GB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
         cluster.adapt(minimum_cores=0, maximum_memory="4GB")
@@ -295,7 +295,7 @@ def test_scale_grouped(loop):
         cores=2,
         memory="2GiB",
         local_directory="/tmp",
-        job_extra=["-V"],
+        job_extra_directives=["-V"],
         loop=loop,
     ) as cluster:
         with Client(cluster) as client:
@@ -352,7 +352,8 @@ def test_config_name_pbs_takes_custom_config():
         "cores": 1,
         "memory": "2 GB",
         "walltime": "00:02",
-        "job-extra": [],
+        "job-extra": None,
+        "job-extra-directives": [],
         "name": "myname",
         "processes": 1,
         "interface": None,
diff --git a/dask_jobqueue/tests/test_sge.py b/dask_jobqueue/tests/test_sge.py
index 3863915..10e2da2 100644
--- a/dask_jobqueue/tests/test_sge.py
+++ b/dask_jobqueue/tests/test_sge.py
@@ -49,7 +49,8 @@ def test_config_name_sge_takes_custom_config():
         "cores": 1,
         "memory": "2 GB",
         "walltime": "00:02",
-        "job-extra": [],
+        "job-extra": None,
+        "job-extra-directives": [],
         "name": "myname",
         "processes": 1,
         "interface": None,
@@ -82,7 +83,7 @@ def test_job_script(tmpdir):
         project="my-project",
         walltime="02:00:00",
         job_script_prologue=["export MY_VAR=my_var"],
-        job_extra=["-w e", "-m e"],
+        job_extra_directives=["-w e", "-m e"],
         log_directory=log_directory,
         resource_spec="h_vmem=12G,mem_req=12G",
     ) as cluster:
diff --git a/dask_jobqueue/tests/test_slurm.py b/dask_jobqueue/tests/test_slurm.py
index 53523c5..ae11e56 100644
--- a/dask_jobqueue/tests/test_slurm.py
+++ b/dask_jobqueue/tests/test_slurm.py
@@ -123,7 +123,7 @@ def test_basic(loop):
         cores=2,
         processes=1,
         memory="2GiB",
-        # job_extra=["-D /"],
+        # job_extra_directives=["-D /"],
         loop=loop,
     ) as cluster:
         with Client(cluster) as client:
@@ -156,7 +156,7 @@ def test_adaptive(loop):
         cores=2,
         processes=1,
         memory="2GB",
-        # job_extra=["-D /"],
+        # job_extra_directives=["-D /"],
         loop=loop,
     ) as cluster:
         cluster.adapt()
@@ -184,7 +184,8 @@ def test_config_name_slurm_takes_custom_config():
         "cores": 1,
         "memory": "2 GB",
         "walltime": "00:02",
-        "job-extra": [],
+        "job-extra": None,
+        "job-extra-directives": [],
         "name": "myname",
         "processes": 1,
         "interface": None,