Repository: spacetelescope/jwst

Issue:
Put output gaia catalog in specified directory
_Issue [JP-3494](https://jira.stsci.edu/browse/JP-3494) was created on JIRA by [Misty Cracraft](https://jira.stsci.edu/secure/ViewProfile.jspa?name=cracraft):_

When running the pipeline in a notebook in one directory, pointing to data in another directory, the tweakreg step puts the output gaia (or other source) catalog in the directory where the notebook is run, not the directory where the data is running. Specifying the output_dir as part of the tweakreg step should tell the pipeline to put all output into that directory. This ticket is to ask for an update in the pipeline code to have the abs_refcat code check the output_dir path and send output to that specified directory. This prevents the full run of a notebook due to the need to manually move the file from one location to another before it can be finished.

Discussion:
emolter: started regression tests [here](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1356/)
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/spacetelescope/jwst/pull/8386?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 77.33%. Comparing base [(`2fb073e`)](https://app.codecov.io/gh/spacetelescope/jwst/commit/2fb073e828d06f18ef7bd163445007c463b45136?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) to head [(`3b4787b`)](https://app.codecov.io/gh/spacetelescope/jwst/pull/8386?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).
> Report is 14 commits behind head on master.

> :exclamation: Current head 3b4787b differs from pull request most recent head 8884850. Consider uploading reports for the commit 8884850 to get more accurate results


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #8386      +/-   ##
==========================================
+ Coverage   75.31%   77.33%   +2.02%     
==========================================
  Files         474      451      -23     
  Lines       38965    35049    -3916     
==========================================
- Hits        29345    27106    -2239     
+ Misses       9620     7943    -1677     
```

| [Flag](https://app.codecov.io/gh/spacetelescope/jwst/pull/8386/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [nightly](https://app.codecov.io/gh/spacetelescope/jwst/pull/8386/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) | `77.33% <ø> (+<0.01%)` | :arrow_up: | Carriedforward from [2fb073e](https://app.codecov.io/gh/spacetelescope/jwst/commit/2fb073e828d06f18ef7bd163445007c463b45136?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) |

*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope) to find out more.


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/spacetelescope/jwst/pull/8386?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=spacetelescope).

emolter: new regtest run [started here](https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1365/)
hbushouse: Failures in latest regtest run are all benign and unrelated to this PR.

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index baee14669..de15c485f 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -20,6 +20,11 @@ ramp_fitting
   to use uint16 instead of uint8, in order to avoid potential
   overflow/wraparound problems. [#8377]
 
+tweakreg
+--------
+
+- Output source catalog file now respects ``output_dir`` parameter. [#8386]
+
 
 1.14.0 (2024-03-29)
 ===================
diff --git a/jwst/tweakreg/tweakreg_step.py b/jwst/tweakreg/tweakreg_step.py
index fa46c52b0..8971010d4 100644
--- a/jwst/tweakreg/tweakreg_step.py
+++ b/jwst/tweakreg/tweakreg_step.py
@@ -243,23 +243,7 @@ class TweakRegStep(Step):
                               .format(len(catalog), filename))
 
             if new_cat and self.save_catalogs:
-                catalog_filename = filename.replace(
-                    '.fits', '_cat.{}'.format(self.catalog_format)
-                )
-                if self.catalog_format == 'ecsv':
-                    fmt = 'ascii.ecsv'
-                elif self.catalog_format == 'fits':
-                    # NOTE: The catalog must not contain any 'None' values.
-                    #       FITS will also not clobber existing files.
-                    fmt = 'fits'
-                else:
-                    raise ValueError(
-                        '\'catalog_format\' must be "ecsv" or "fits".'
-                    )
-                catalog.write(catalog_filename, format=fmt, overwrite=True)
-                self.log.info('Wrote source catalog: {}'
-                              .format(catalog_filename))
-                image_model.meta.tweakreg_catalog = catalog_filename
+                image_model = self._write_catalog(image_model, catalog, filename)
 
             # Temporarily attach catalog to the image model so that it follows
             # the grouping by exposure, to be removed after use below
@@ -412,7 +396,10 @@ class TweakRegStep(Step):
             #        whatever convention is determined by the JWST Cal Working
             #        Group.
             if self.save_abs_catalog:
-                output_name = 'fit_{}_ref.ecsv'.format(self.abs_refcat.lower())
+                if self.output_dir is None:
+                    output_name = 'fit_{}_ref.ecsv'.format(self.abs_refcat.lower())
+                else:
+                    output_name = path.join(self.output_dir, 'fit_{}_ref.ecsv'.format(self.abs_refcat.lower()))
             else:
                 output_name = None
 
@@ -524,6 +511,55 @@ class TweakRegStep(Step):
                     )
 
         return images
+    
+
+    def _write_catalog(self, image_model, catalog, filename):
+        '''
+        Determine output filename for catalog based on outfile for step
+        and output dir, then write catalog to file.
+
+        Parameters
+        ----------
+        image_model : jwst.datamodels.ImageModel
+            Image model containing the source catalog.
+        catalog : astropy.table.Table
+            Table containing the source catalog.
+        filename : str
+            Output filename for step
+
+        Returns
+        -------
+        image_model : jwst.datamodels.ImageModel
+            Image model with updated catalog information.
+        '''
+
+        catalog_filename = str(filename).replace(
+                    '.fits', '_cat.{}'.format(self.catalog_format)
+                )
+        if self.catalog_format == 'ecsv':
+            fmt = 'ascii.ecsv'
+        elif self.catalog_format == 'fits':
+            # NOTE: The catalog must not contain any 'None' values.
+            #       FITS will also not clobber existing files.
+            fmt = 'fits'
+        else:
+            raise ValueError(
+                '\'catalog_format\' must be "ecsv" or "fits".'
+            )
+        if self.output_dir is None:
+            catalog.write(catalog_filename, format=fmt, overwrite=True)
+        else:
+            catalog.write(
+                path.join(self.output_dir, catalog_filename),
+                format=fmt,
+                overwrite=True
+            )
+        self.log.info('Wrote source catalog: {}'
+                      .format(catalog_filename))
+        image_model.meta.tweakreg_catalog = catalog_filename
+
+        return image_model
+
 
     def _is_wcs_correction_small(self, wcs, twcs):
         """Check that the newly tweaked wcs hasn't gone off the rails"""


Test patch:
diff --git a/jwst/tweakreg/tests/test_tweakreg.py b/jwst/tweakreg/tests/test_tweakreg.py
index cac427d95..a240fca28 100644
--- a/jwst/tweakreg/tests/test_tweakreg.py
+++ b/jwst/tweakreg/tests/test_tweakreg.py
@@ -3,6 +3,7 @@ import os
 
 import asdf
 from astropy.modeling.models import Shift
+from astropy.table import Table
 import pytest
 
 from jwst.tweakreg import tweakreg_step
@@ -10,6 +11,16 @@ from jwst.tweakreg import tweakreg_catalog
 from stdatamodels.jwst.datamodels import ImageModel
 
 
+@pytest.fixture
+def dummy_source_catalog():
+
+    columns = ['id', 'xcentroid', 'ycentroid', 'flux']
+    catalog = Table(names=columns, dtype=(int, float, float, float))
+    catalog.add_row([1, 100.0, 100.0, 100.0])
+
+    return catalog
+
+
 @pytest.mark.parametrize("offset, is_good", [(1 / 3600, True), (11 / 3600, False)])
 def test_is_wcs_correction_small(offset, is_good):
     path = os.path.join(os.path.dirname(__file__), "mosaic_long_i2d_gwcs.asdf")
@@ -63,3 +74,19 @@ def test_expected_failure_bad_starfinder():
     model = ImageModel()
     with pytest.raises(ValueError):
         tweakreg_catalog.make_tweakreg_catalog(model, 5.0, bkg_boxsize=400, starfinder='bad_value')
+
+
+def test_write_catalog(dummy_source_catalog, tmp_cwd):
+    '''
+    Covers an issue where catalog write did not respect self.output_dir
+    '''
+
+    OUTDIR = 'outdir'
+    model = ImageModel()
+    step = tweakreg_step.TweakRegStep()
+    os.mkdir(OUTDIR)
+    step.output_dir = OUTDIR
+    expected_outfile = os.path.join(OUTDIR, 'catalog.ecsv')
+    step._write_catalog(model, dummy_source_catalog, 'catalog.ecsv')
+
+    assert os.path.exists(expected_outfile)
\ No newline at end of file