Repository: esheldon/metadetect
Pull request #223: Refactor top-level LSST metadetect
Author: login: arunkannawadi
Created: 2024-10-21T19:46:38Z | Merged: 2025-03-20T19:07:34Z

This is a preview of the refactoring I mentioned in #220 . It currently refactors only the top-most level making run_metadetect, which is the typical entry point, a wrapper around a Task. The extensive coverage with unit tests should make us all feel confident going about such refactoring. The fact that the the unit tests pass without any changes also shows that the public APIs remain unchanged.


Diff:
diff --git a/.github/workflows/lsst-tests-ngmix-upstream.yml b/.github/workflows/lsst-tests-ngmix-upstream.yml
index f7060013..5e3fb7af 100644
--- a/.github/workflows/lsst-tests-ngmix-upstream.yml
+++ b/.github/workflows/lsst-tests-ngmix-upstream.yml
@@ -38,8 +38,8 @@ jobs:
           conda install --quiet --yes --file requirements.txt
           conda install --quiet --yes --file dev-requirements.txt
           conda install -q -y stackvana=0 lsstdesc-weaklensingdeblending
+          conda install lsstdesc-wl-shear-sims
 
-          pip install --no-deps git+https://github.com/LSSTDESC/descwl-shear-sims.git
           pip install --no-deps git+https://github.com/LSSTDESC/descwl_coadd.git
 
           python -m pip install -e .
diff --git a/.github/workflows/lsst-tests.yml b/.github/workflows/lsst-tests.yml
index a242e9a7..2c64f3f2 100644
--- a/.github/workflows/lsst-tests.yml
+++ b/.github/workflows/lsst-tests.yml
@@ -38,8 +38,8 @@ jobs:
           conda install --quiet --yes --file requirements.txt
           conda install --quiet --yes --file dev-requirements.txt
           conda install -q -y stackvana=0 lsstdesc-weaklensingdeblending
+          conda install lsstdesc-wl-shear-sims
 
-          pip install --no-deps git+https://github.com/LSSTDESC/descwl-shear-sims.git
           pip install --no-deps git+https://github.com/LSSTDESC/descwl_coadd.git
 
           python -m pip install -e .
diff --git a/metadetect/lsst/README.md b/metadetect/lsst/README.md
new file mode 100644
index 00000000..5000b0e7
--- /dev/null
+++ b/metadetect/lsst/README.md
@@ -0,0 +1,20 @@
+# Metadetect for Rubin Observatory / LSST
+
+This subpackage contains code for running metadetect on data structures and algorithms available from the Rubin Science Pipelines.
+
+## Config framework
+
+### Overview of the config framework in Rubin Science Pipelines
+The algorithms within Rubin Science Pipelines are implemented as `Task`s and `PipelineTask`s, each with an associate `Config` object.
+This architecture allows using the same algorithm in multiple places, configured appropriately as required.
+Thus, a `Config` object is initially mutable.
+The default values are specified in the individual fields and they can be overridden in a special method called `setDefaults`.
+However, once the `freeze` method is called, it becomes immutable (via public methods) and the values cannot be changed any further.
+After a `Config` instance is frozen, its `validate` method is called to check that the configuration is valid one.
+This is intended to check the data types and relationships between the fields.
+When run as part of the pipelines, the pipeline execution (or `pex`) in Rubin Science Pipelines will call these methods before beginning the execution and persist them along with the processing runs for reproducibility.
+
+### Use of Config framework in metadetect
+The `metadetect` algorithm are also implemented as `Task`s with thin-wrappers around them for backwards compatibility.
+The `Task` entry points are intended to be used in production, whereas the top-level wrapper functions continue to serve as entry points for on-the-fly simulations with other packages (e.g., [descwl-shear-sims](https://github.com/LSSTDESC/descwl-shear-sims) and [descwl_coadd](https://github.com/LSSTDESC/descwl_coadd)).
+In order to prevent changes to the default values upstream going unnoticed and affecting the performance of `metadetect`, the unit tests in `tests/test_lsst_dm_configs.py` check that the `Config` objects what they have been validated on simulations.
diff --git a/metadetect/lsst/measure.py b/metadetect/lsst/measure.py
index 3baba38f..4959a29c 100644
--- a/metadetect/lsst/measure.py
+++ b/metadetect/lsst/measure.py
@@ -5,12 +5,16 @@
 import ngmix
 
 import lsst.afw.table as afw_table
-from lsst.meas.algorithms import SourceDetectionTask, SourceDetectionConfig
+from lsst.meas.algorithms import SourceDetectionTask
+from lsst.meas.algorithms import SourceDetectionConfig as OriginalSourceDetectionConfig
 from lsst.meas.deblender import SourceDeblendTask, SourceDeblendConfig
 from lsst.meas.base import (
     SingleFrameMeasurementConfig,
     SingleFrameMeasurementTask,
 )
+from lsst.pex.config import Config, ConfigurableField, Field
+from lsst.pipe.base import Task
+import lsst.afw.image as afw_image
 import lsst.geom as geom
 from lsst.pex.exceptions import (
     InvalidParameterError,
@@ -23,7 +27,7 @@
 from ..fitting import fit_mbobs_wavg, get_wavg_output_struct
 
 from . import util
-from .util import ContextNoiseReplacer
+from .util import ContextNoiseReplacer, get_jacobian
 from . import vis
 from .defaults import DEFAULT_THRESH
 
@@ -32,11 +36,164 @@
 LOG = logging.getLogger('lsst_measure')
 
 
+class SourceDetectionConfig(OriginalSourceDetectionConfig):
+    @property
+    def thresh(self):
+        return self.thresholdValue
+
+    @thresh.setter
+    def thresh(self, value):
+        self.thresholdValue = value
+
+
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
+
+
+class DetectAndDeblendConfig(Config):
+    meas = ConfigurableField[SingleFrameMeasurementConfig](
+        doc="Measurement config",
+        target=SingleFrameMeasurementTask,
+    )
+
+    detect = ConfigurableField[SourceDetectionConfig](
+        doc="Detection config",
+        target=SourceDetectionTask
+    )
+
+    deblend = ConfigurableField[SourceDeblendConfig](
+        doc="Deblend config",
+        target=SourceDeblendTask
+    )
+
+    seed = Field[int](
+        doc="Random rng seed",
+        default=42,
+    )
+
+    def setDefaults(self):
+        super().setDefaults()
+
+        # defaults for measurement config
+        self.meas.plugins.names = [
+            "base_SdssCentroid",
+            "base_PsfFlux",
+            "base_SkyCoord",
+        ]
+
+        # set these slots to none because we aren't running these algorithms
+        self.meas.slots.apFlux = None
+        self.meas.slots.gaussianFlux = None
+        self.meas.slots.calibFlux = None
+        self.meas.slots.modelFlux = None
+
+        # goes with SdssShape above
+        self.meas.slots.shape = None
+
+        # fix odd issue where it things things are near the edge
+        self.meas.plugins['base_SdssCentroid'].binmax = 1
+
+        # defaults for detection config
+        # DM does not have config default stability.  Set all of them explicitly
+        self.detect.minPixels = 1
+        self.detect.isotropicGrow = True
+        self.detect.combinedGrow = True
+        self.detect.nSigmaToGrow = 2.4
+        self.detect.returnOriginalFootprints = False
+        self.detect.includeThresholdMultiplier = 1.0
+        self.detect.thresholdPolarity = "positive"
+        self.detect.adjustBackground = 0.0
+        # these are ignored since we are doing reEstimateBackground = False
+        # detection_config.background
+        # detection_config.tempLocalBackground
+        # detection_config.doTempLocalBackground
+        # detection_config.tempWideBackground
+        # detection_config.doTempWideBackground
+        self.detect.nPeaksMaxSimple = 1
+        self.detect.nSigmaForKernel = 7.0
+        self.detect.excludeMaskPlanes = util.get_detection_mask()
+
+        # the defaults changed from from stdev to pixel_std but
+        # we don't want that
+
+        self.detect.thresholdType = "stdev"
+        # our changes from defaults
+        self.detect.reEstimateBackground = False
+
+        self.detect.thresholdValue = DEFAULT_THRESH
+
+        # these will be ignored when finding the image standard deviation
+        self.detect.statsMask = util.get_stats_mask()
+
+        # deblend config
+        # these tasks must use the same schema and all be constructed before any
+        # other tasks using the same schema are run because schema is modified in
+        # place by tasks, and the constructor does a check that fails if we do this
+        # afterward
+        self.deblend.maxFootprintArea = 0
+
+
+class DetectAndDeblendTask(Task):
+    ConfigClass = DetectAndDeblendConfig
+    _DefaultName = "detect_and_deblend"
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        schema = afw_table.SourceTable.makeMinimalSchema()
+        self.makeSubtask("meas", schema=schema)
+        self.makeSubtask("detect", schema=schema)
+        self.makeSubtask("deblend", schema=schema)
+        self.rng = np.random.RandomState(seed=self.config.seed)
+
+    def run(self, mbexp, show=False):
+        if len(mbexp.singles) > 1:
+            detexp = util.coadd_exposures(mbexp.singles)
+            if detexp is None:
+                return [], None
+        else:
+            detexp = mbexp.singles[0]
+
+        # background measurement within the detection code requires ExposureF
+        if not isinstance(detexp, afw_image.ExposureF):
+            detexp = afw_image.ExposureF(detexp, deep=True)
+
+        schema = self.deblend.schema  # should be the same for all tasks
+        afw_table.CoordKey.addErrorFields(schema)
+        table = afw_table.SourceTable.make(schema)
+        result = self.detect.run(table, detexp)
+
+        if result is not None:
+            sources = result.sources
+            self.deblend.run(detexp, sources)
+
+            with ContextNoiseReplacer(
+                detexp, sources, self.rng, config=self.meas.config.noiseReplacer
+            ) as replacer:
+
+                for source in sources:
+
+                    if source.get('deblend_nChild') != 0:
+                        continue
+
+                    source_id = source.getId()
+
+                    with replacer.sourceInserted(source_id):
+                        self.meas.callMeasure(source, detexp)
+
+        else:
+            sources = []
+
+        if show:
+            vis.show_exp(detexp, use_mpl=True, sources=sources)
+
+        return sources, detexp
+
+
 def detect_and_deblend(
     mbexp,
-    rng,
+    rng=None,
     thresh=DEFAULT_THRESH,
     show=False,
+    config=None,
 ):
     """
     run detection and deblending of peaks, as well as basic measurments such as
@@ -62,123 +219,23 @@ def detect_and_deblend(
     sources, detexp
         The sources and the detection exposure
     """
-    import lsst.afw.image as afw_image
+    config_override = config if config is not None else {}
+    if thresh:
+        if 'detect' not in config_override:
+            config_override['detect'] = {}
+        config_override['detect']['thresholdValue'] = thresh
 
-    if len(mbexp.singles) > 1:
-        detexp = util.coadd_exposures(mbexp.singles)
-        if detexp is None:
-            return [], None
-    else:
-        detexp = mbexp.singles[0]
+    config = DetectAndDeblendConfig()
+    config.setDefaults()
 
-    # background measurement within the detection code requires ExposureF
-    if not isinstance(detexp, afw_image.ExposureF):
-        detexp = afw_image.ExposureF(detexp, deep=True)
-
-    schema = afw_table.SourceTable.makeMinimalSchema()
-
-    # Setup algorithms to run
-    meas_config = SingleFrameMeasurementConfig()
-    meas_config.plugins.names = [
-        "base_SdssCentroid",
-        "base_PsfFlux",
-        "base_SkyCoord",
-    ]
-
-    # set these slots to none because we aren't running these algorithms
-    meas_config.slots.apFlux = None
-    meas_config.slots.gaussianFlux = None
-    meas_config.slots.calibFlux = None
-    meas_config.slots.modelFlux = None
-
-    # goes with SdssShape above
-    meas_config.slots.shape = None
-
-    # fix odd issue where it things things are near the edge
-    meas_config.plugins['base_SdssCentroid'].binmax = 1
-
-    meas_task = SingleFrameMeasurementTask(
-        config=meas_config,
-        schema=schema,
-    )
-    # avoids a warning spamming for every object
-    afw_table.CoordKey.addErrorFields(schema)
-
-    detection_config = SourceDetectionConfig()
-
-    # DM does not have config default stability.  Set all of them explicitly
-    detection_config.minPixels = 1
-    detection_config.isotropicGrow = True
-    detection_config.combinedGrow = True
-    detection_config.nSigmaToGrow = 2.4
-    detection_config.returnOriginalFootprints = False
-    detection_config.includeThresholdMultiplier = 1.0
-    detection_config.thresholdPolarity = "positive"
-    detection_config.adjustBackground = 0.0
-    detection_config.reEstimateBackground = True
-    # these are ignored since we are doing reEstimateBackground = False
-    # detection_config.background
-    # detection_config.tempLocalBackground
-    # detection_config.doTempLocalBackground
-    # detection_config.tempWideBackground
-    # detection_config.doTempWideBackground
-
-    detection_config.nPeaksMaxSimple = 1
-    detection_config.nSigmaForKernel = 7.0
-    detection_config.excludeMaskPlanes = util.get_detection_mask(detexp)
-
-    # the defaults changed from from stdev to pixel_std but
-    # we don't want that
-
-    detection_config.thresholdType = "stdev"
-    # our changes from defaults
-    detection_config.reEstimateBackground = False
-
-    detection_config.thresholdValue = thresh
-
-    # these will be ignored when finding the image standard deviation
-    detection_config.statsMask = util.get_stats_mask(detexp)
-
-    detection_task = SourceDetectionTask(config=detection_config)
-
-    # these tasks must use the same schema and all be constructed before any
-    # other tasks using the same schema are run because schema is modified in
-    # place by tasks, and the constructor does a check that fails if we do this
-    # afterward
-    deblend_config = SourceDeblendConfig()
-    deblend_config.maxFootprintArea = 0
-    deblend_task = SourceDeblendTask(
-        config=deblend_config,
-        schema=schema,
-    )
+    util.override_config(config, config_override)
 
-    table = afw_table.SourceTable.make(schema)
-
-    result = detection_task.run(table, detexp)
-
-    if result is not None:
-        sources = result.sources
-        deblend_task.run(detexp, sources)
-
-        with ContextNoiseReplacer(detexp, sources, rng) as replacer:
-
-            for source in sources:
-
-                if source.get('deblend_nChild') != 0:
-                    continue
-
-                source_id = source.getId()
-
-                with replacer.sourceInserted(source_id):
-                    meas_task.callMeasure(source, detexp)
-
-    else:
-        sources = []
-
-    if show:
-        vis.show_exp(detexp, use_mpl=True, sources=sources)
-
-    return sources, detexp
+    config.freeze()
+    config.validate()
+    task = DetectAndDeblendTask(config=config)
+    if rng is not None:
+        task.rng = rng
+    return task.run(mbexp, show)
 
 
 def measure(
@@ -594,7 +651,6 @@ def _extract_jacobian_at_source(exp, source):
     Jacobian: ngmix.Jacobian
         The local jacobian
     """
-    from .util import get_jacobian
 
     orig_cen = exp.getWcs().skyToPixel(source.getCoord())
 
@@ -698,8 +754,6 @@ def get_output(wcs, source, res, bmask, stamp_size, exp_bbox):
     ndarray
         Has the fields from res, with new fields added, see get_output_dtype
     """
-    import lsst.afw.image as afw_image
-
     output = get_output_struct(res)
 
     orig_cen = source.getCentroid()
diff --git a/metadetect/lsst/metadetect.py b/metadetect/lsst/metadetect.py
index 3ebe193c..cebe39b4 100644
--- a/metadetect/lsst/metadetect.py
+++ b/metadetect/lsst/metadetect.py
@@ -2,15 +2,32 @@
 import numpy as np
 import ngmix
 from ngmix.gexceptions import BootPSFFailure
+from lsst.pex.config import (
+    Config,
+    ChoiceField,
+    ConfigField,
+    ConfigurableField,
+    Field,
+    FieldValidationError,
+    ListField,
+)
+from lsst.pipe.base import Task
+from lsst.meas.algorithms import SourceDetectionTask
 
 from .. import procflags
 
 from .skysub import subtract_sky_mbexp
 
-from .configs import get_config
+from .defaults import (
+    DEFAULT_FWHM_REG,
+    DEFAULT_FWHM_SMOOTH,
+    DEFAULT_STAMP_SIZES,
+    DEFAULT_SUBTRACT_SKY,
+    DEFAULT_WEIGHT_FWHMS,
+)
 from . import measure
 from .metacal_exposures import get_metacal_mbexps_fixnoise
-from .util import get_integer_center, get_jacobian
+from .util import get_integer_center, get_jacobian, override_config
 
 LOG = logging.getLogger('lsst_metadetect')
 
@@ -58,50 +75,217 @@ def run_metadetect(
         metacal_psf is set to 'fitgauss' and the fitting fails
     """
 
-    config = get_config(config)
+    config_override = config if config is not None else {}
+    config = MetadetectConfig()
+    config.setDefaults()
 
-    ormask = combine_ormasks(mbexp, ormasks)
-    mfrac, wgts = get_mfrac_mbexp(mbexp=mbexp, mfrac_mbexp=mfrac_mbexp)
+    override_config(config, config_override)
 
-    if config['subtract_sky']:
-        subtract_sky_mbexp(mbexp=mbexp, thresh=config['detect']['thresh'])
+    config.freeze()
+    config.validate()
+    task = MetadetectTask(config=config)
+    result = task.run(mbexp, noise_mbexp, rng, mfrac_mbexp, ormasks, show=show,)
+    return result
 
-    psf_stats = fit_original_psfs_mbexp(
-        mbexp=mbexp,
-        wgts=wgts,
-        rng=rng,
+
+class WeightConfig(Config):
+    fwhm = Field[float](
+        doc="FWHM of the Gaussian weight function (in arcseconds)",
+        default=None,
+    )
+    fwhm_smooth = Field[float](
+        doc="FWHM of the Gaussian smoothing function (in arcseconds)",
+        default=DEFAULT_FWHM_SMOOTH,
+    )
+    fwhm_reg = Field[float](
+        doc="FWHM of the Gaussian regularization function (in arcseconds)",
+        default=DEFAULT_FWHM_REG,
+    )
+
+
+class LmParseConfig(Config):
+    maxfev = Field[int](
+        doc="Maximum number of function evaluations",
+        default=4000,
+    )
+    ftol = Field[float](
+        doc="Relative error desired in the sum of squares",
+        default=1.0e-5,
+    )
+    xtol = Field[float](
+        doc="Relative error desired in the approximate solution",
+        default=1.0e-5,
+    )
+
+
+class PsfConfig(Config):
+    """Config class for fitting original PSFs."""
+    model = ChoiceField[str](
+        doc="Model for fitting original PSFs",
+        default="am",
+        allowed={
+            "am": "Adaptive moments",  # TODO: What are the other possible values?
+            "gauss": "Gaussian",
+        },
+        optional=False,
+    )
+    ntry = Field[int](
+        doc="Number of tries for fitting original PSFs",
+        default=4,
+    )
+    lm_pars = ConfigField[LmParseConfig](
+        doc="Config for Levenberg-Marquardt fitting",
+    )
+
+
+class MetacalConfig(Config):
+    use_noise_image = Field[bool](
+        doc="Whether to use the noise image",
+        default=True,
+    )
+    psf = ChoiceField[str](
+        doc="PSF model",
+        default="fitgauss",
+        allowed={
+            "fitgauss": "Fit a Gaussian to the PSF",
+        }
+    )
+    types = ListField[str](
+        doc="List of artificial shears to apply.",
+        default=["noshear", "1p", "1m",],
+    )
+
+    def validate(self):
+        super().validate()
+        if not set(self.types).issubset({"noshear", "1p", "1m", "2p", "2m"}):
+            raise FieldValidationError(
+                self.__class__.types,
+                self,
+                "types must be a list consisting of any combinations of "
+                "{'noshear', '1p', '1m', '2p', '2m'}",
+            )
+
+
+class MetadetectConfig(Config):
+    subtract_sky = Field[bool](
+        doc="Whether to subtract the sky before running metadetect",
+        default=DEFAULT_SUBTRACT_SKY,
+    )
+    meas_type = ChoiceField[str](
+        doc="Measurement type",
+        default="wmom",
+        allowed={
+            "wmom": "Weighted moments",
+            "ksigma": "Fourier moments",  # TODO: improve docstring
+            "pgauss": "Pre-seeing moments",
+            "am": "Adaptive moments",
+        }
     )
 
-    fitter = get_fitter(config, rng=rng)
+    weight = ConfigField[WeightConfig](
+        doc="FWHM of the Gaussian weight function (in arcseconds)",
+    )
 
-    metacal_types = config['metacal'].get('types', None)
+    psf = ConfigField[PsfConfig](
+        doc="Config for fitting the original PSFs",
+    )
 
-    mdict, noise_mdict = get_metacal_mbexps_fixnoise(
-        mbexp=mbexp, noise_mbexp=noise_mbexp, types=metacal_types,
+    detect = ConfigurableField(
+        doc="Detection config",
+        target=SourceDetectionTask,
     )
+    metacal = ConfigField[MetacalConfig](
+        doc="Config for metacal",
+    )
+
+    @property
+    def stamp_size(self):
+        # Caution: This provides a backdoor entry to change the behavior
+        # by changing the DEFAULT_STAMP_SIZES.
+        return DEFAULT_STAMP_SIZES[self.meas_type]
+
+    def setDefaults(self):
+        super().setDefaults()
+        self.weight.fwhm = DEFAULT_WEIGHT_FWHMS.get(self.meas_type, None)
+
+    def validate(self):
+        super().validate()
+
+        if self.stamp_size != DEFAULT_STAMP_SIZES[self.meas_type]:
+            raise FieldValidationError(
+                self.__class__.stamp_size,
+                self,
+                f"stamp_size must be {DEFAULT_STAMP_SIZES[self.meas_type]} "
+                f"for meas_type {self.meas_type}",
+            )
+
+
+class MetadetectTask(Task):
+    ConfigClass = MetadetectConfig
+    _DefaultName = "metadetect"
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.makeSubtask("detect")
 
-    result = {}
-    for shear_str, mcal_mbexp in mdict.items():
+    def run(
+        self,
+        mbexp,
+        noise_mbexp,
+        rng,
+        mfrac_mbexp=None,
+        ormasks=None,
+        show=False,
+    ):
 
-        res = detect_deblend_and_measure(
-            mbexp=mcal_mbexp,
-            fitter=fitter,
-            config=config,
+        # This is to support methods that are not yet refactored.
+        config = self.config.toDict()
+        # Because this is a property and not a Field, we set this explicitly.
+        config['stamp_size'] = self.config.stamp_size
+        config['detect']['thresh'] = self.detect.config.thresholdValue
+
+        ormask = combine_ormasks(mbexp, ormasks)
+        mfrac, wgts = get_mfrac_mbexp(mbexp=mbexp, mfrac_mbexp=mfrac_mbexp)
+
+        if self.config.subtract_sky:
+            subtract_sky_mbexp(mbexp=mbexp, thresh=self.config.detect.thresholdValue)
+
+        psf_stats = fit_original_psfs_mbexp(
+            mbexp=mbexp,
+            wgts=wgts,
             rng=rng,
-            show=show,
         )
 
-        if res is not None:
-            band = mcal_mbexp.filters[0]
-            exp = mcal_mbexp[band]
+        fitter = get_fitter(config, rng=rng)
 
-            add_mfrac(config=config, mfrac=mfrac, res=res, exp=exp)
-            add_ormask(ormask, res)
-            add_original_psf(psf_stats, res)
+        metacal_types = config['metacal'].get('types', None)
 
-        result[shear_str] = res
+        mdict, noise_mdict = get_metacal_mbexps_fixnoise(
+            mbexp=mbexp, noise_mbexp=noise_mbexp, types=metacal_types,
+        )
 
-    return result
+        result = {}
+        for shear_str, mcal_mbexp in mdict.items():
+
+            res = detect_deblend_and_measure(
+                mbexp=mcal_mbexp,
+                fitter=fitter,
+                config=config,
+                rng=rng,
+                show=show,
+            )
+
+            if res is not None:
+                band = mcal_mbexp.filters[0]
+                exp = mcal_mbexp[band]
+
+                add_mfrac(config=config, mfrac=mfrac, res=res, exp=exp)
+                add_ormask(ormask, res)
+                add_original_psf(psf_stats, res)
+
+            result[shear_str] = res
+
+        return result
 
 
 def detect_deblend_and_measure(
@@ -419,7 +603,6 @@ def fit_original_psfs_mbexp(mbexp, rng, wgts):
 
     This can fail and flags will be set, but we proceed
     """
-    from .measure import extract_psf_image
 
     assert len(wgts) == len(mbexp)
     wsum = sum(wgts)
@@ -445,7 +628,7 @@ def fit_original_psfs_mbexp(mbexp, rng, wgts):
             )
             jac = get_jacobian(exp=exp, cen=cen)
 
-            psf_im = extract_psf_image(exp, cen)
+            psf_im = measure.extract_psf_image(exp, cen)
 
             psf_cen = (np.array(psf_im.shape)-1.0)/2.0
             psf_jacob = jac.copy()
diff --git a/metadetect/lsst/skysub.py b/metadetect/lsst/skysub.py
index 53f36f80..91de57e8 100644
--- a/metadetect/lsst/skysub.py
+++ b/metadetect/lsst/skysub.py
@@ -3,13 +3,114 @@
     SubtractBackgroundTask,
     SubtractBackgroundConfig,
     SourceDetectionTask,
-    SourceDetectionConfig,
 )
+from lsst.pex.config import Config, ConfigurableField, Field
+from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
+from lsst.pipe.base import Task, TaskError
 
 from .defaults import DEFAULT_THRESH
+from .measure import SourceDetectionConfig
 from . import util
 
 
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
+
+
+class IterateDetectionSkySubConfig(Config):
+    niter = Field[int](
+        doc="Number of iterations",
+        default=2,
+    )
+
+    detect = ConfigurableField[SourceDetectionConfig](
+        doc="Detection config",
+        target=SourceDetectionTask
+    )
+
+    back = ConfigurableField[SubtractBackgroundConfig](
+        doc="Subtract background config",
+        target=SubtractBackgroundTask
+    )
+
+    def setDefaults(self):
+        super().setDefaults()
+
+        # detection
+        self.detect.reEstimateBackground = False
+        self.detect.thresholdValue = DEFAULT_THRESH
+
+        # background subtraction
+        bp_to_skip = util.get_stats_mask()
+        self.back = SubtractBackgroundConfig(ignoredPixelMask=bp_to_skip)
+
+
+class IterateDetectionSkySubTask(Task):
+    ConfigClass = IterateDetectionSkySubConfig
+    _DefaultName = "iterate_detection_and_skysub"
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.makeSubtask("detect")
+        self.makeSubtask("back")
+
+    def run(self, exposure):
+        if self.config.niter < 1:
+            raise ValueError(f'niter {self.config.niter} is less than 1')
+
+        schema = afw_table.SourceTable.makeMinimalSchema()
+        table = afw_table.SourceTable.make(schema)
+
+        # keep a running sum of each sky that was subtracted
+        try:
+            sky_meas = 0.0
+            for i in range(self.config.niter):
+                self.back.run(exposure)
+                result = self.detect.run(table, exposure)
+
+                sky_meas += exposure.getMetadata()['BGMEAN']
+
+            meta = exposure.getMetadata()
+
+            # this is the overall sky we subtracted in all iterations
+            meta['BGMEAN'] = sky_meas
+        except LSSTRuntimeError as err:
+            err = str(err).replace('lsst::pex::exceptions::RuntimeError:', '')
+            self.detect.log.warn(err)
+            result = None
+        except TaskError as err:
+            err = str(err).replace('lsst.pipe.base.task.TaskError:', '')
+            self.detect.log.warn(err)
+            result = None
+
+        return result
+
+
+class SubtractSkyMbExpConfig(Config):
+    iterate_detection_and_skysub = ConfigurableField[IterateDetectionSkySubConfig](
+        doc="Iterate detection and sky subtraction config",
+        target=IterateDetectionSkySubTask
+    )
+
+    def setDefaults(self):
+        super().setDefaults()
+
+        self.iterate_detection_and_skysub.detect.thresholdValue = DEFAULT_THRESH
+        self.iterate_detection_and_skysub.niter = 2
+
+
+class SubtractSkyMbExpTask(Task):
+    ConfigClass = SubtractSkyMbExpConfig
+    _DefaultName = "subtract_sky_mbexp"
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.makeSubtask("iterate_detection_and_skysub")
+
+    def run(self, mbexp):
+        for exp in mbexp:
+            self.iterate_detection_and_skysub.run(exposure=exp)
+
+
 def determine_and_subtract_sky(exp):
     """
     Determine and subtract the sky from the input exposure.
@@ -30,7 +131,7 @@ def determine_and_subtract_sky(exp):
     return background
 
 
-def subtract_sky_mbexp(mbexp, thresh=DEFAULT_THRESH):
+def subtract_sky_mbexp(mbexp, thresh=DEFAULT_THRESH, config=None):
     """
     subtract sky
 
@@ -41,15 +142,30 @@ def subtract_sky_mbexp(mbexp, thresh=DEFAULT_THRESH):
     thresh: float
         Threshold for detection
     """
-    for exp in mbexp:
-        iterate_detection_and_skysub(
-            exposure=exp,
-            thresh=thresh,
-        )
+    config_override = config if config is not None else {}
+
+    if thresh:
+        if 'iterate_detection_and_skysub' not in config_override:
+            config_override['iterate_detection_and_skysub'] = {}
+        if 'detect' not in config_override['iterate_detection_and_skysub']:
+            config_override['iterate_detection_and_skysub']['detect'] = {}
+
+        detect_override = config_override['iterate_detection_and_skysub']['detect']
+        detect_override['thresholdValue'] = thresh
+
+    config = SubtractSkyMbExpConfig()
+    config.setDefaults()
+
+    util.override_config(config, config_override)
+
+    config.freeze()
+    config.validate()
+    task = SubtractSkyMbExpTask(config=config)
+    task.run(mbexp=mbexp)
 
 
 def iterate_detection_and_skysub(
-    exposure, thresh, niter=2,
+    exposure, thresh, niter=2, config=None
 ):
     """
     Iterate detection and sky subtraction
@@ -68,39 +184,24 @@ def iterate_detection_and_skysub(
     -------
     Result from running the detection task
     """
-    from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
-    from lsst.pipe.base.task import TaskError
     if niter < 1:
         raise ValueError(f'niter {niter} is less than 1')
 
-    schema = afw_table.SourceTable.makeMinimalSchema()
-    detection_config = SourceDetectionConfig()
-    detection_config.reEstimateBackground = False
-    detection_config.thresholdValue = thresh
-    detection_task = SourceDetectionTask(config=detection_config)
-
-    table = afw_table.SourceTable.make(schema)
+    config_override = config if config is not None else {}
 
-    # keep a running sum of each sky that was subtracted
-    try:
-        sky_meas = 0.0
-        for i in range(niter):
-            determine_and_subtract_sky(exposure)
-            result = detection_task.run(table, exposure)
+    # set threshold
+    if 'detect' not in config_override:
+        config_override['detect'] = {}
+    config_override['detect']['thresholdValue'] = thresh
 
-            sky_meas += exposure.getMetadata()['BGMEAN']
+    if niter:
+        config_override['niter'] = niter
 
-        meta = exposure.getMetadata()
+    config = IterateDetectionSkySubConfig()
+    config.setDefaults()
 
-        # this is the overall sky we subtracted in all iterations
-        meta['BGMEAN'] = sky_meas
-    except LSSTRuntimeError as err:
-        err = str(err).replace('lsst::pex::exceptions::RuntimeError:', '')
-        detection_task.log.warn(err)
-        result = None
-    except TaskError as err:
-        err = str(err).replace('lsst.pipe.base.task.TaskError:', '')
-        detection_task.log.warn(err)
-        result = None
+    util.override_config(config, config_override)
 
+    task = IterateDetectionSkySubTask(config=config)
+    result = task.run(exposure)
     return result
diff --git a/metadetect/lsst/util.py b/metadetect/lsst/util.py
index 3e9a8b5e..4e0218c8 100644
--- a/metadetect/lsst/util.py
+++ b/metadetect/lsst/util.py
@@ -25,7 +25,7 @@ class ContextNoiseReplacer(object):
         # do something
     """
 
-    def __init__(self, exposure, sources, rng, noise_image=None):
+    def __init__(self, exposure, sources, rng, config=None, noise_image=None):
         from lsst.meas.base import NoiseReplacerConfig, NoiseReplacer
         import lsst.afw.detection as afw_det
 
@@ -39,11 +39,12 @@ def __init__(self, exposure, sources, rng, noise_image=None):
         # making the field the same for all metacal images we reduce variance in
         # the calculation of the response
 
-        config = NoiseReplacerConfig()
+        if config is None:
+            config = NoiseReplacerConfig()
 
-        # TODO DM needs to fix the crash
-        # config.noiseSource = 'variance'
-        config.noiseSeedMultiplier = rng.randint(0, 2**24)
+            # TODO DM needs to fix the crash
+            # config.noiseSource = 'variance'
+            config.noiseSeedMultiplier = rng.randint(0, 2**24)
 
         if noise_image is None:
             # TODO remove_poisson should be true for real data
@@ -797,17 +798,11 @@ def get_integer_center(wcs, bbox, as_double=False):
     return pixcen, skycen
 
 
-def get_stats_mask(exp):
+def get_stats_mask(*args, **kwargs):
     """
-    Get a stats mask for use in getting image statistics.  If BRIGHT
-    is found in the mask plane it is added to the usual
+    Get a stats mask for use in getting image statistics.
 
-    ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA']
-
-    Parameters
-    ----------
-    exp: lsst.afw.image.ExposureF
-        The exposure
+    ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA', 'BRIGHT']
 
     Returns
     -------
@@ -817,39 +812,32 @@ def get_stats_mask(exp):
     # these will be ignored when finding the image standard deviation
     # stats_mask = ['BAD', 'SAT', 'EDGE', 'NO_DATA']
 
-    stats_mask = ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA']
-
-    if 'BRIGHT' in exp.mask.getMaskPlaneDict():
-        stats_mask += ['BRIGHT']
+    import lsst.afw.image as afw_image
+    afw_image.Mask.addMaskPlane('BRIGHT')
+    stats_mask = ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA', 'BRIGHT']
 
     return stats_mask
 
 
-def get_detection_mask(exp=None):
+def get_detection_mask(*args, **kwargs):
     """
     Get a mask for detection. Regions with these flags set will not be searched
     for objects.
 
-    Default is ['EDGE', 'NO_DATA']
+    Default is ['EDGE', 'NO_DATA', 'BRIGHT]
 
     BRIGHT is defined in descwl_coadd, so it is not guaranteed to be present in
-    the global mask bit space.  Only add BRIGHT to detection mask BRIGHT if it
-    is found in the global mask plane dict
-
-    Parameters
-    ----------
-    exp: lsst.afw.image.ExposureF, optional
-        The exposure
+    the global mask bit space.  We add BRIGHT to detection mask. If it already
+    exists, it does not add a new plane and is harmless.
 
     Returns
     -------
     A list of mask planes
     """
 
-    mask = ['EDGE', 'NO_DATA']
-
-    if exp is not None and 'BRIGHT' in exp.mask.getMaskPlaneDict():
-        mask += ['BRIGHT']
+    import lsst.afw.image as afw_image
+    afw_image.Mask.addMaskPlane('BRIGHT')
+    mask = ['EDGE', 'NO_DATA', 'BRIGHT']
 
     return mask
 
@@ -905,3 +893,31 @@ def extract_multiband_coadd_data(coadd_data_list):
         'mfrac_mbexp': mfrac_mbexp,
         'ormasks': ormasks,
     }
+
+
+def override_config(config, config_override: dict):
+    """Override the values in a Config instance with those in a dict.
+
+    The `config` object can have arbitrary level of nesting, and to override
+    the values, `config_override` should have the same nested structure.
+    To allow for arbitrary nesting, this function calls itself recursively.
+
+    Parameters
+    ----------
+    config : `lsst.pex.config.Config`
+        The configuration object to override.
+    config_override : `dict`
+        The dictionary of values to override the configuration with.
+
+    Raises
+    ------
+    AttributeError
+        Raised if a key in `config_override` is not an attribute of `config`.
+    """
+    for key, value in config_override.items():
+        if isinstance(value, dict):
+            subconfig = getattr(config, key)
+            # Use recursion to allow for arbitrarily nested config objects.
+            override_config(subconfig, value)
+        else:
+            setattr(config, key, value)


Reviews:
user: login: ismael-mendoza
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2024-10-31T16:10:40Z
user: login: beckermr
body: Why were the ngmix upstream tests removed?

Ditto for the extra test skeips etc. All of that needs to be backed out before we can consider merging this.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2024-10-31T20:43:59Z
user: login: beckermr
body: I have some comments/questions. Sorry for the delay!
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-02-11T12:37:51Z
user: login: arunkannawadi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T17:42:45Z
user: login: arunkannawadi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T23:01:02Z
user: login: arunkannawadi
body: I made changes/addressed the questions from previous review and I think this is ready for another round of review, Matt.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T23:04:43Z
user: login: beckermr
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T17:26:53Z
user: login: beckermr
body: LGTM on the code! One more request.

Can we add some documentation somewhere or links to it to explain a bit about how the config works, how we can ensure the config is not changed, etc.? 

We had a ton of discussion on this, but I forgot the results and it feels like we should write it down.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T17:30:11Z
user: login: beckermr
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-20T16:23:05Z

Review comments:
diff_hunk: @@ -32,11 +38,162 @@
 LOG = logging.getLogger('lsst_measure')
 
 
+class DetectAndDeblendConfig(Config):
+    meas = ConfigurableField[SingleFrameMeasurementConfig](
+        doc="Measuremetn config",
path: metadetect/lsst/measure.py
user: login: ismael-mendoza
body: ```suggestion
        doc="Measurement config",
```
created_at: 2024-10-31T16:10:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,11 +36,165 @@
 LOG = logging.getLogger('lsst_measure')
 
 
+class SourceDetectionConfig(OriginalSourceDetectionConfig):
+    @property
+    def thresh(self):
+        return self.thresholdValue
+
+    @thresh.setter
+    def thresh(self, value):
+        self.thresholdValue = value
+
+
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
+
+
+class DetectAndDeblendConfig(Config):
+    meas = ConfigurableField[SingleFrameMeasurementConfig](
+        doc="Measurement config",
+        target=SingleFrameMeasurementTask,
+    )
+
+    detect = ConfigurableField[SourceDetectionConfig](
+        doc="Detection config",
+        target=SourceDetectionTask
+    )
+
+    deblend = ConfigurableField[SourceDeblendConfig](
+        doc="Deblend config",
+        target=SourceDeblendTask
+    )
+
+    seed = Field[int](
+        doc="Random rng seed",
+        default=42,
+    )
+
+    def setDefaults(self):
+        super().setDefaults()
+
+        # defaults for measurement config
+        self.meas.plugins.names = [
+            "base_SdssCentroid",
+            "base_PsfFlux",
+            "base_SkyCoord",
+        ]
+
+        # set these slots to none because we aren't running these algorithms
+        self.meas.slots.apFlux = None
+        self.meas.slots.gaussianFlux = None
+        self.meas.slots.calibFlux = None
+        self.meas.slots.modelFlux = None
+
+        # goes with SdssShape above
+        self.meas.slots.shape = None
+
+        # fix odd issue where it things things are near the edge
+        self.meas.plugins['base_SdssCentroid'].binmax = 1
+
+        # defaults for detection config
+        # DM does not have config default stability.  Set all of them explicitly
+        self.detect.minPixels = 1
+        self.detect.isotropicGrow = True
+        self.detect.combinedGrow = True
+        self.detect.nSigmaToGrow = 2.4
+        self.detect.returnOriginalFootprints = False
+        self.detect.includeThresholdMultiplier = 1.0
+        self.detect.thresholdPolarity = "positive"
+        self.detect.adjustBackground = 0.0
+        self.detect.reEstimateBackground = True
path: metadetect/lsst/measure.py
user: login: beckermr
body: This one is listed twice, see below.
created_at: 2025-02-11T12:28:10Z
author_association: COLLABORATOR
diff_hunk: @@ -32,11 +36,165 @@
 LOG = logging.getLogger('lsst_measure')
 
 
+class SourceDetectionConfig(OriginalSourceDetectionConfig):
+    @property
+    def thresh(self):
+        return self.thresholdValue
+
+    @thresh.setter
+    def thresh(self, value):
+        self.thresholdValue = value
+
+
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
+
+
+class DetectAndDeblendConfig(Config):
+    meas = ConfigurableField[SingleFrameMeasurementConfig](
+        doc="Measurement config",
+        target=SingleFrameMeasurementTask,
+    )
+
+    detect = ConfigurableField[SourceDetectionConfig](
+        doc="Detection config",
+        target=SourceDetectionTask
+    )
+
+    deblend = ConfigurableField[SourceDeblendConfig](
+        doc="Deblend config",
+        target=SourceDeblendTask
+    )
+
+    seed = Field[int](
+        doc="Random rng seed",
+        default=42,
+    )
+
+    def setDefaults(self):
+        super().setDefaults()
+
+        # defaults for measurement config
+        self.meas.plugins.names = [
+            "base_SdssCentroid",
+            "base_PsfFlux",
+            "base_SkyCoord",
+        ]
+
+        # set these slots to none because we aren't running these algorithms
+        self.meas.slots.apFlux = None
+        self.meas.slots.gaussianFlux = None
+        self.meas.slots.calibFlux = None
+        self.meas.slots.modelFlux = None
+
+        # goes with SdssShape above
+        self.meas.slots.shape = None
+
+        # fix odd issue where it things things are near the edge
+        self.meas.plugins['base_SdssCentroid'].binmax = 1
+
+        # defaults for detection config
+        # DM does not have config default stability.  Set all of them explicitly
+        self.detect.minPixels = 1
+        self.detect.isotropicGrow = True
+        self.detect.combinedGrow = True
+        self.detect.nSigmaToGrow = 2.4
+        self.detect.returnOriginalFootprints = False
+        self.detect.includeThresholdMultiplier = 1.0
+        self.detect.thresholdPolarity = "positive"
+        self.detect.adjustBackground = 0.0
+        self.detect.reEstimateBackground = True
+        # these are ignored since we are doing reEstimateBackground = False
+        # detection_config.background
+        # detection_config.tempLocalBackground
+        # detection_config.doTempLocalBackground
+        # detection_config.tempWideBackground
+        # detection_config.doTempWideBackground
+        self.detect.nPeaksMaxSimple = 1
+        self.detect.nSigmaForKernel = 7.0
+        self.detect.excludeMaskPlanes = util.get_detection_mask()
+
+        # the defaults changed from from stdev to pixel_std but
+        # we don't want that
+
+        self.detect.thresholdType = "stdev"
+        # our changes from defaults
+        self.detect.reEstimateBackground = False
path: metadetect/lsst/measure.py
user: login: beckermr
body: This is the other entry.
created_at: 2025-02-11T12:28:22Z
author_association: COLLABORATOR
diff_hunk: @@ -58,50 +75,217 @@ def run_metadetect(
         metacal_psf is set to 'fitgauss' and the fitting fails
     """
 
-    config = get_config(config)
+    config_override = config if config is not None else {}
+    config = MetadetectConfig()
+    config.setDefaults()
 
-    ormask = combine_ormasks(mbexp, ormasks)
-    mfrac, wgts = get_mfrac_mbexp(mbexp=mbexp, mfrac_mbexp=mfrac_mbexp)
+    override_config(config, config_override)
 
-    if config['subtract_sky']:
-        subtract_sky_mbexp(mbexp=mbexp, thresh=config['detect']['thresh'])
+    config.freeze()
+    config.validate()
+    task = MetadetectTask(config=config)
+    result = task.run(mbexp, noise_mbexp, rng, mfrac_mbexp, ormasks, show=show,)
+    return result
 
-    psf_stats = fit_original_psfs_mbexp(
-        mbexp=mbexp,
-        wgts=wgts,
-        rng=rng,
+
+class WeightConfig(Config):
+    fwhm = Field[float](
+        doc="FWHM of the Gaussian weight function (in pixel units)",
+        default=None,
+    )
+    fwhm_smooth = Field[float](
+        doc="FWHM of the Gaussian smoothing function (in pixel units)",
+        default=DEFAULT_FWHM_SMOOTH,
+    )
+    fwhm_reg = Field[float](
+        doc="FWHM of the Gaussian regularization function (in pixel units)",
+        default=DEFAULT_FWHM_REG,
+    )
path: metadetect/lsst/metadetect.py
user: login: beckermr
body: I think the units here are arcseconds?
created_at: 2025-02-11T12:31:19Z
author_association: COLLABORATOR
diff_hunk: @@ -3,13 +3,124 @@
     SubtractBackgroundTask,
     SubtractBackgroundConfig,
     SourceDetectionTask,
-    SourceDetectionConfig,
 )
+from lsst.meas.algorithms import SourceDetectionConfig as OriginalSourceDetectionConfig
+from lsst.pex.config import Config, ConfigurableField, Field
+from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
+from lsst.pipe.base import Task, TaskError
 
 from .defaults import DEFAULT_THRESH
 from . import util
 
 
+class SourceDetectionConfig(OriginalSourceDetectionConfig):
+    @property
+    def thresholdValue(self):
+        return self.thresholdValue
+
+    @thresholdValue.setter
+    def thresholdValue(self, value):
+        self.thresholdValue = value
+
+
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
path: metadetect/lsst/skysub.py
user: login: beckermr
body: This bit of config boilerplate is repeated from the measure module. If one instance of this code needs to be changed, does the other one need to be changed too? If yes, then we should have this bit of boilerplate appear only once.
created_at: 2025-02-11T12:34:08Z
author_association: COLLABORATOR
diff_hunk: @@ -68,39 +194,35 @@ def iterate_detection_and_skysub(
     -------
     Result from running the detection task
     """
-    from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
-    from lsst.pipe.base.task import TaskError
     if niter < 1:
         raise ValueError(f'niter {niter} is less than 1')
 
-    schema = afw_table.SourceTable.makeMinimalSchema()
-    detection_config = SourceDetectionConfig()
-    detection_config.reEstimateBackground = False
-    detection_config.thresholdValue = thresh
-    detection_task = SourceDetectionTask(config=detection_config)
+    config_override = config if config is not None else {}
 
-    table = afw_table.SourceTable.make(schema)
+    # set threshold
+    if 'detect' not in config_override:
+        config_override['detect'] = {}
+    config_override['detect']['thresholdValue'] = thresh
 
-    # keep a running sum of each sky that was subtracted
-    try:
-        sky_meas = 0.0
-        for i in range(niter):
-            determine_and_subtract_sky(exposure)
-            result = detection_task.run(table, exposure)
+    if niter:
+        config_override['niter'] = niter
 
-            sky_meas += exposure.getMetadata()['BGMEAN']
+    config = IterateDetectionSkySubConfig()
+    config.setDefaults()
 
-        meta = exposure.getMetadata()
-
-        # this is the overall sky we subtracted in all iterations
-        meta['BGMEAN'] = sky_meas
-    except LSSTRuntimeError as err:
-        err = str(err).replace('lsst::pex::exceptions::RuntimeError:', '')
-        detection_task.log.warn(err)
-        result = None
-    except TaskError as err:
-        err = str(err).replace('lsst.pipe.base.task.TaskError:', '')
-        detection_task.log.warn(err)
-        result = None
+    util.override_config(config, config_override)
 
+    task = IterateDetectionSkySubTask(config=config)
+    result = task.run(exposure)
     return result
+
+
+class SkySubtractionTask(Task):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.makeSubtask("subtract_background")
+        self.makeSubtask("detect_sources")
+
+    def run(self, exposure, **kwargs):
+        return self._sky_sub(exposure, **kwargs)
path: metadetect/lsst/skysub.py
user: login: beckermr
body: Where is this method defined? It doesn't look like this should be attached to the Task class. Does it happen on-the-fly after that calls to makeSubtask?
created_at: 2025-02-11T12:35:56Z
author_association: COLLABORATOR
diff_hunk: @@ -817,39 +812,32 @@ def get_stats_mask(exp):
     # these will be ignored when finding the image standard deviation
     # stats_mask = ['BAD', 'SAT', 'EDGE', 'NO_DATA']
 
-    stats_mask = ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA']
-
-    if 'BRIGHT' in exp.mask.getMaskPlaneDict():
-        stats_mask += ['BRIGHT']
+    import lsst.afw.image as afw_image
+    afw_image.Mask.addMaskPlane('BRIGHT')
+    stats_mask = ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA', 'BRIGHT']
path: metadetect/lsst/util.py
user: login: beckermr
body: There are changes in the logic here around BRIGHT handling. Are those inteneded?
created_at: 2025-02-11T12:37:09Z
author_association: COLLABORATOR
diff_hunk: @@ -58,50 +75,217 @@ def run_metadetect(
         metacal_psf is set to 'fitgauss' and the fitting fails
     """
 
-    config = get_config(config)
+    config_override = config if config is not None else {}
+    config = MetadetectConfig()
+    config.setDefaults()
 
-    ormask = combine_ormasks(mbexp, ormasks)
-    mfrac, wgts = get_mfrac_mbexp(mbexp=mbexp, mfrac_mbexp=mfrac_mbexp)
+    override_config(config, config_override)
 
-    if config['subtract_sky']:
-        subtract_sky_mbexp(mbexp=mbexp, thresh=config['detect']['thresh'])
+    config.freeze()
+    config.validate()
+    task = MetadetectTask(config=config)
+    result = task.run(mbexp, noise_mbexp, rng, mfrac_mbexp, ormasks, show=show,)
+    return result
 
-    psf_stats = fit_original_psfs_mbexp(
-        mbexp=mbexp,
-        wgts=wgts,
-        rng=rng,
+
+class WeightConfig(Config):
+    fwhm = Field[float](
+        doc="FWHM of the Gaussian weight function (in pixel units)",
+        default=None,
+    )
+    fwhm_smooth = Field[float](
+        doc="FWHM of the Gaussian smoothing function (in pixel units)",
+        default=DEFAULT_FWHM_SMOOTH,
+    )
+    fwhm_reg = Field[float](
+        doc="FWHM of the Gaussian regularization function (in pixel units)",
+        default=DEFAULT_FWHM_REG,
+    )
+
+
+class LmParseConfig(Config):
+    maxfev = Field[int](
+        doc="Maximum number of function evaluations",
+        default=4000,
+    )
+    ftol = Field[float](
+        doc="Relative error desired in the sum of squares",
+        default=1.0e-5,
+    )
+    xtol = Field[float](
+        doc="Relative error desired in the approximate solution",
+        default=1.0e-5,
+    )
+
+
+class PsfConfig(Config):
+    """Config class for fitting original PSFs."""
+    model = ChoiceField[str](
+        doc="Model for fitting original PSFs",
+        default="am",
+        allowed={
+            "am": "Adaptive moments",  # TODO: What are the other possible values?
+            "gauss": "Gaussian",
+        },
+        optional=False,
+    )
+    ntry = Field[int](
+        doc="Number of tries for fitting original PSFs",
+        default=4,
+    )
+    lm_pars = ConfigField[LmParseConfig](
+        doc="Config for Levenberg-Marquardt fitting",
+    )
+
+
+class MetacalConfig(Config):
+    use_noise_image = Field[bool](
+        doc="Whether to use the noise image",
+        default=True,
+    )
+    psf = ChoiceField[str](
+        doc="PSF model",
+        default="fitgauss",
+        allowed={
+            "fitgauss": "Fit a Gaussian to the PSF",
+        }
+    )
+    types = ListField[str](
+        doc="List of artificial shears to apply.",
+        default=["noshear", "1p", "1m",],
+    )
+
+    def validate(self):
+        super().validate()
+        if not set(self.types).issubset({"noshear", "1p", "1m", "2p", "2m"}):
+            raise FieldValidationError(
+                self.__class__.types,
+                self,
+                "types must be a list consisting of any combinations of "
+                "{'noshear', '1p', '1m', '2p', '2m'}",
+            )
+
+
+class MetadetectConfig(Config):
+    subtract_sky = Field[bool](
+        doc="Whether to subtract the sky before running metadetect",
+        default=DEFAULT_SUBTRACT_SKY,
+    )
+    meas_type = ChoiceField[str](
+        doc="Measurement type",
+        default="wmom",
+        allowed={
+            "wmom": "Weighted moments",
+            "ksigma": "Fourier moments",  # TODO: improve docstring
+            "pgauss": "Pre-seeing moments",
+            "am": "Adaptive moments",
+        }
     )
 
-    fitter = get_fitter(config, rng=rng)
+    weight = ConfigField[WeightConfig](
+        doc="FWHM of the Gaussian weight function (in pixel units)",
path: metadetect/lsst/metadetect.py
user: login: arunkannawadi
body: I believe this is also in arcseconds?
created_at: 2025-03-03T16:55:03Z
author_association: COLLABORATOR
diff_hunk: @@ -32,11 +36,165 @@
 LOG = logging.getLogger('lsst_measure')
 
 
+class SourceDetectionConfig(OriginalSourceDetectionConfig):
+    @property
+    def thresh(self):
+        return self.thresholdValue
+
+    @thresh.setter
+    def thresh(self, value):
+        self.thresholdValue = value
+
+
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
+
+
+class DetectAndDeblendConfig(Config):
+    meas = ConfigurableField[SingleFrameMeasurementConfig](
+        doc="Measurement config",
+        target=SingleFrameMeasurementTask,
+    )
+
+    detect = ConfigurableField[SourceDetectionConfig](
+        doc="Detection config",
+        target=SourceDetectionTask
+    )
+
+    deblend = ConfigurableField[SourceDeblendConfig](
+        doc="Deblend config",
+        target=SourceDeblendTask
+    )
+
+    seed = Field[int](
+        doc="Random rng seed",
+        default=42,
+    )
+
+    def setDefaults(self):
+        super().setDefaults()
+
+        # defaults for measurement config
+        self.meas.plugins.names = [
+            "base_SdssCentroid",
+            "base_PsfFlux",
+            "base_SkyCoord",
+        ]
+
+        # set these slots to none because we aren't running these algorithms
+        self.meas.slots.apFlux = None
+        self.meas.slots.gaussianFlux = None
+        self.meas.slots.calibFlux = None
+        self.meas.slots.modelFlux = None
+
+        # goes with SdssShape above
+        self.meas.slots.shape = None
+
+        # fix odd issue where it things things are near the edge
+        self.meas.plugins['base_SdssCentroid'].binmax = 1
+
+        # defaults for detection config
+        # DM does not have config default stability.  Set all of them explicitly
+        self.detect.minPixels = 1
+        self.detect.isotropicGrow = True
+        self.detect.combinedGrow = True
+        self.detect.nSigmaToGrow = 2.4
+        self.detect.returnOriginalFootprints = False
+        self.detect.includeThresholdMultiplier = 1.0
+        self.detect.thresholdPolarity = "positive"
+        self.detect.adjustBackground = 0.0
+        self.detect.reEstimateBackground = True
path: metadetect/lsst/measure.py
user: login: arunkannawadi
body: Right, this followed straight from the existing code where it is first set to `True` and then `False`. The last one would be the one to stand, so removing this line.
created_at: 2025-03-03T16:58:17Z
author_association: COLLABORATOR
diff_hunk: @@ -68,39 +194,35 @@ def iterate_detection_and_skysub(
     -------
     Result from running the detection task
     """
-    from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
-    from lsst.pipe.base.task import TaskError
     if niter < 1:
         raise ValueError(f'niter {niter} is less than 1')
 
-    schema = afw_table.SourceTable.makeMinimalSchema()
-    detection_config = SourceDetectionConfig()
-    detection_config.reEstimateBackground = False
-    detection_config.thresholdValue = thresh
-    detection_task = SourceDetectionTask(config=detection_config)
+    config_override = config if config is not None else {}
 
-    table = afw_table.SourceTable.make(schema)
+    # set threshold
+    if 'detect' not in config_override:
+        config_override['detect'] = {}
+    config_override['detect']['thresholdValue'] = thresh
 
-    # keep a running sum of each sky that was subtracted
-    try:
-        sky_meas = 0.0
-        for i in range(niter):
-            determine_and_subtract_sky(exposure)
-            result = detection_task.run(table, exposure)
+    if niter:
+        config_override['niter'] = niter
 
-            sky_meas += exposure.getMetadata()['BGMEAN']
+    config = IterateDetectionSkySubConfig()
+    config.setDefaults()
 
-        meta = exposure.getMetadata()
-
-        # this is the overall sky we subtracted in all iterations
-        meta['BGMEAN'] = sky_meas
-    except LSSTRuntimeError as err:
-        err = str(err).replace('lsst::pex::exceptions::RuntimeError:', '')
-        detection_task.log.warn(err)
-        result = None
-    except TaskError as err:
-        err = str(err).replace('lsst.pipe.base.task.TaskError:', '')
-        detection_task.log.warn(err)
-        result = None
+    util.override_config(config, config_override)
 
+    task = IterateDetectionSkySubTask(config=config)
+    result = task.run(exposure)
     return result
+
+
+class SkySubtractionTask(Task):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.makeSubtask("subtract_background")
+        self.makeSubtask("detect_sources")
+
+    def run(self, exposure, **kwargs):
+        return self._sky_sub(exposure, **kwargs)
path: metadetect/lsst/skysub.py
user: login: arunkannawadi
body: Agree! This looks like a bug. I'll look into it.
created_at: 2025-03-03T17:19:34Z
author_association: COLLABORATOR
diff_hunk: @@ -817,39 +812,32 @@ def get_stats_mask(exp):
     # these will be ignored when finding the image standard deviation
     # stats_mask = ['BAD', 'SAT', 'EDGE', 'NO_DATA']
 
-    stats_mask = ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA']
-
-    if 'BRIGHT' in exp.mask.getMaskPlaneDict():
-        stats_mask += ['BRIGHT']
+    import lsst.afw.image as afw_image
+    afw_image.Mask.addMaskPlane('BRIGHT')
+    stats_mask = ['BAD', 'EDGE', 'DETECTED', 'DETECTED_NEGATIVE', 'NO_DATA', 'BRIGHT']
path: metadetect/lsst/util.py
user: login: arunkannawadi
body: Yes, these are intentional changes but there is no logic change here. The change was required was to keep this function not depend on any specific `Exposure` object `exp` . Instead of looking if `exp` has a `BRIGHT` mask plane and adding to this list, we always add it to the list. If `exp` does not have it defined, then the `addMaskPlane` call above defines it but doesn't set the bit in any pixel and everything proceeds as before. If it has `BRIGHT` plane defined already, then `addMaskPlane` call doesn't do anything and everything proceeds as before as well.
created_at: 2025-03-03T17:40:13Z
author_association: COLLABORATOR
diff_hunk: @@ -3,13 +3,124 @@
     SubtractBackgroundTask,
     SubtractBackgroundConfig,
     SourceDetectionTask,
-    SourceDetectionConfig,
 )
+from lsst.meas.algorithms import SourceDetectionConfig as OriginalSourceDetectionConfig
+from lsst.pex.config import Config, ConfigurableField, Field
+from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
+from lsst.pipe.base import Task, TaskError
 
 from .defaults import DEFAULT_THRESH
 from . import util
 
 
+class SourceDetectionConfig(OriginalSourceDetectionConfig):
+    @property
+    def thresholdValue(self):
+        return self.thresholdValue
+
+    @thresholdValue.setter
+    def thresholdValue(self, value):
+        self.thresholdValue = value
+
+
+SourceDetectionTask.ConfigClass = SourceDetectionConfig
path: metadetect/lsst/skysub.py
user: login: arunkannawadi
body: Good catch! This seems to have gotten duplicated during the refactoring. Keeping the one in `measure`. The `ConfigClass` has to be set again here though since it's imported from `meas_algorithms`.
created_at: 2025-03-06T22:57:28Z
author_association: COLLABORATOR
diff_hunk: @@ -68,39 +194,35 @@ def iterate_detection_and_skysub(
     -------
     Result from running the detection task
     """
-    from lsst.pex.exceptions import RuntimeError as LSSTRuntimeError
-    from lsst.pipe.base.task import TaskError
     if niter < 1:
         raise ValueError(f'niter {niter} is less than 1')
 
-    schema = afw_table.SourceTable.makeMinimalSchema()
-    detection_config = SourceDetectionConfig()
-    detection_config.reEstimateBackground = False
-    detection_config.thresholdValue = thresh
-    detection_task = SourceDetectionTask(config=detection_config)
+    config_override = config if config is not None else {}
 
-    table = afw_table.SourceTable.make(schema)
+    # set threshold
+    if 'detect' not in config_override:
+        config_override['detect'] = {}
+    config_override['detect']['thresholdValue'] = thresh
 
-    # keep a running sum of each sky that was subtracted
-    try:
-        sky_meas = 0.0
-        for i in range(niter):
-            determine_and_subtract_sky(exposure)
-            result = detection_task.run(table, exposure)
+    if niter:
+        config_override['niter'] = niter
 
-            sky_meas += exposure.getMetadata()['BGMEAN']
+    config = IterateDetectionSkySubConfig()
+    config.setDefaults()
 
-        meta = exposure.getMetadata()
-
-        # this is the overall sky we subtracted in all iterations
-        meta['BGMEAN'] = sky_meas
-    except LSSTRuntimeError as err:
-        err = str(err).replace('lsst::pex::exceptions::RuntimeError:', '')
-        detection_task.log.warn(err)
-        result = None
-    except TaskError as err:
-        err = str(err).replace('lsst.pipe.base.task.TaskError:', '')
-        detection_task.log.warn(err)
-        result = None
+    util.override_config(config, config_override)
 
+    task = IterateDetectionSkySubTask(config=config)
+    result = task.run(exposure)
     return result
+
+
+class SkySubtractionTask(Task):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.makeSubtask("subtract_background")
+        self.makeSubtask("detect_sources")
+
+    def run(self, exposure, **kwargs):
+        return self._sky_sub(exposure, **kwargs)
path: metadetect/lsst/skysub.py
user: login: arunkannawadi
body: This turned out to be some extra bit of lines that weren't being used anywhere. Removed them.
created_at: 2025-03-06T23:01:01Z
author_association: COLLABORATOR
diff_hunk: @@ -58,50 +75,217 @@ def run_metadetect(
         metacal_psf is set to 'fitgauss' and the fitting fails
     """
 
-    config = get_config(config)
+    config_override = config if config is not None else {}
+    config = MetadetectConfig()
+    config.setDefaults()
 
-    ormask = combine_ormasks(mbexp, ormasks)
-    mfrac, wgts = get_mfrac_mbexp(mbexp=mbexp, mfrac_mbexp=mfrac_mbexp)
+    override_config(config, config_override)
 
-    if config['subtract_sky']:
-        subtract_sky_mbexp(mbexp=mbexp, thresh=config['detect']['thresh'])
+    config.freeze()
+    config.validate()
+    task = MetadetectTask(config=config)
+    result = task.run(mbexp, noise_mbexp, rng, mfrac_mbexp, ormasks, show=show,)
+    return result
 
-    psf_stats = fit_original_psfs_mbexp(
-        mbexp=mbexp,
-        wgts=wgts,
-        rng=rng,
+
+class WeightConfig(Config):
+    fwhm = Field[float](
+        doc="FWHM of the Gaussian weight function (in pixel units)",
+        default=None,
+    )
+    fwhm_smooth = Field[float](
+        doc="FWHM of the Gaussian smoothing function (in pixel units)",
+        default=DEFAULT_FWHM_SMOOTH,
+    )
+    fwhm_reg = Field[float](
+        doc="FWHM of the Gaussian regularization function (in pixel units)",
+        default=DEFAULT_FWHM_REG,
+    )
+
+
+class LmParseConfig(Config):
+    maxfev = Field[int](
+        doc="Maximum number of function evaluations",
+        default=4000,
+    )
+    ftol = Field[float](
+        doc="Relative error desired in the sum of squares",
+        default=1.0e-5,
+    )
+    xtol = Field[float](
+        doc="Relative error desired in the approximate solution",
+        default=1.0e-5,
+    )
+
+
+class PsfConfig(Config):
+    """Config class for fitting original PSFs."""
+    model = ChoiceField[str](
+        doc="Model for fitting original PSFs",
+        default="am",
+        allowed={
+            "am": "Adaptive moments",  # TODO: What are the other possible values?
+            "gauss": "Gaussian",
+        },
+        optional=False,
+    )
+    ntry = Field[int](
+        doc="Number of tries for fitting original PSFs",
+        default=4,
+    )
+    lm_pars = ConfigField[LmParseConfig](
+        doc="Config for Levenberg-Marquardt fitting",
+    )
+
+
+class MetacalConfig(Config):
+    use_noise_image = Field[bool](
+        doc="Whether to use the noise image",
+        default=True,
+    )
+    psf = ChoiceField[str](
+        doc="PSF model",
+        default="fitgauss",
+        allowed={
+            "fitgauss": "Fit a Gaussian to the PSF",
+        }
+    )
+    types = ListField[str](
+        doc="List of artificial shears to apply.",
+        default=["noshear", "1p", "1m",],
+    )
+
+    def validate(self):
+        super().validate()
+        if not set(self.types).issubset({"noshear", "1p", "1m", "2p", "2m"}):
+            raise FieldValidationError(
+                self.__class__.types,
+                self,
+                "types must be a list consisting of any combinations of "
+                "{'noshear', '1p', '1m', '2p', '2m'}",
+            )
+
+
+class MetadetectConfig(Config):
+    subtract_sky = Field[bool](
+        doc="Whether to subtract the sky before running metadetect",
+        default=DEFAULT_SUBTRACT_SKY,
+    )
+    meas_type = ChoiceField[str](
+        doc="Measurement type",
+        default="wmom",
+        allowed={
+            "wmom": "Weighted moments",
+            "ksigma": "Fourier moments",  # TODO: improve docstring
+            "pgauss": "Pre-seeing moments",
+            "am": "Adaptive moments",
+        }
     )
 
-    fitter = get_fitter(config, rng=rng)
+    weight = ConfigField[WeightConfig](
+        doc="FWHM of the Gaussian weight function (in pixel units)",
path: metadetect/lsst/metadetect.py
user: login: beckermr
body: yes!
created_at: 2025-03-10T17:26:53Z
author_association: COLLABORATOR

PR comments:
user: login: arunkannawadi
created_at: 2024-11-01T00:23:36Z
body: Thanks to @ismael-mendoza for doing the big chunk of this work. We've tested this by also running the shear recovery tests in `descwl-shear-sims` (shear_test_descwl/test_shear_meas) and that seems to pass.
author_association: COLLABORATOR
user: login: arunkannawadi
created_at: 2025-01-31T22:34:52Z
body: @esheldon @beckermr - have you had a chance to go through this PR?
author_association: COLLABORATOR
user: login: esheldon
created_at: 2025-02-10T19:47:32Z
body: This looks OK to me.  I think we need to run it at scale to stress test it, and look at outputs for sanity checks.
author_association: OWNER
user: login: arunkannawadi
created_at: 2025-02-10T20:59:21Z
body: I am happy to run extensive tests but would need some handholding to get started.
author_association: COLLABORATOR
user: login: arunkannawadi
created_at: 2025-03-20T15:58:50Z
body: I've added a README to the `lsst` subpackage explaining some of this. Let me know what else you'd like to see.

Additionally, I'd like to include `py3.12` to the matrix test (not necessarily in this PR though) but it looks like it couldn't find consistent set of dependencies. GitHub Actions log [here](https://github.com/esheldon/metadetect/actions/runs/13710064653/job/38344479994).
author_association: COLLABORATOR
user: login: beckermr
created_at: 2025-03-20T16:04:37Z
body: Looks like sep is not on numpy2 yet and the feedstock is falling behind. I am working on it now.
author_association: COLLABORATOR
user: login: arunkannawadi
created_at: 2025-03-20T16:08:29Z
body: Thank you! However, the LSST-specific tests, which is where I tried adding 3.12 should not require `sep` at all in the first place. It looks like it's trying to install everything in `requirements.txt` when only a subset of them are required for the `lsst` subpackage. `sxdes` is another one that's not required.
author_association: COLLABORATOR
user: login: beckermr
created_at: 2025-03-20T16:10:56Z
body: Ah we can fix that as well by separating out the files. Are you adding 3.12 here or in another PR?
author_association: COLLABORATOR
user: login: arunkannawadi
created_at: 2025-03-20T16:13:37Z
body: I will add 3.12 in another PR.
author_association: COLLABORATOR
user: login: beckermr
created_at: 2025-03-20T16:22:46Z
body: @arunkannawadi I can help there to separate out the requirements if needed!
author_association: COLLABORATOR
user: login: arunkannawadi
created_at: 2025-03-20T16:24:29Z
body: Alright, thanks. That'll be in a week or so after this week's `stackvana` is out. I don't have the permissions to merge the PR, so I'll leave that to you and Erin.
author_association: COLLABORATOR
user: login: beckermr
created_at: 2025-03-20T16:25:55Z
body: @esheldon you ok to merge here?
author_association: COLLABORATOR