try:<EOL><INDENT>formatinfo = get_lcformat(lcformat,<EOL>use_lcformat_dir=lcformatdir)<EOL>if formatinfo:<EOL><INDENT>(fileglob, readerfunc,<EOL>dtimecols, dmagcols, derrcols,<EOL>magsarefluxes, normfunc) = formatinfo<EOL><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>")<EOL>return None<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION("<STR_LIT>")<EOL>return None<EOL><DEDENT>if timecols is None:<EOL><INDENT>timecols = dtimecols<EOL><DEDENT>if magcols is None:<EOL><INDENT>magcols = dmagcols<EOL><DEDENT>if errcols is None:<EOL><INDENT>errcols = derrcols<EOL><DEDENT>outdict = {}<EOL>for t, m, e in zip(timecols, magcols, errcols):<EOL><INDENT>tasks = [(x, t, m, e, externalparams, lcformat, lcformatdir,<EOL>epdsmooth_sigclip, epdsmooth_windowsize,<EOL>epdsmooth_func, epdsmooth_extraparams) for<EOL>x in lclist]<EOL>pool = mp.Pool(nworkers, maxtasksperchild=maxworkertasks)<EOL>results = pool.map(parallel_epd_worker, tasks)<EOL>pool.close()<EOL>pool.join()<EOL>outdict[m] = results<EOL><DEDENT>return outdict<EOL>

This applies EPD in parallel to all LCs in the input list.

    Parameters
    ----------

    lclist : list of str
        This is the list of light curve files to run EPD on.

    externalparams : dict or None
        This is a dict that indicates which keys in the lcdict obtained from the
        lcfile correspond to the required external parameters. As with timecol,
        magcol, and errcol, these can be simple keys (e.g. 'rjd') or compound
        keys ('magaperture1.mags'). The dict should look something like::

          {'fsv':'<lcdict key>' array: S values for each observation,
           'fdv':'<lcdict key>' array: D values for each observation,
           'fkv':'<lcdict key>' array: K values for each observation,
           'xcc':'<lcdict key>' array: x coords for each observation,
           'ycc':'<lcdict key>' array: y coords for each observation,
           'bgv':'<lcdict key>' array: sky background for each observation,
           'bge':'<lcdict key>' array: sky background err for each observation,
           'iha':'<lcdict key>' array: hour angle for each observation,
           'izd':'<lcdict key>' array: zenith distance for each observation}

        Alternatively, if these exact keys are already present in the lcdict,
        indicate this by setting externalparams to None.

    timecols,magcols,errcols : lists of str
        The keys in the lcdict produced by your light curve reader function that
        correspond to the times, mags/fluxes, and associated measurement errors
        that will be used as inputs to the EPD process. If these are None, the
        default values for `timecols`, `magcols`, and `errcols` for your light
        curve format will be used here.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curve files.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    epdsmooth_sigclip : float or int or sequence of two floats/ints or None
        This specifies how to sigma-clip the input LC before fitting the EPD
        function to it.

        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    epdsmooth_windowsize : int
        This is the number of LC points to smooth over to generate a smoothed
        light curve that will be used to fit the EPD function.

    epdsmooth_func : Python function
        This sets the smoothing filter function to use. A Savitsky-Golay filter
        is used to smooth the light curve by default. The functions that can be
        used with this kwarg are listed in `varbase.trends`. If you want to use
        your own function, it MUST have the following signature::

                def smoothfunc(mags_array, window_size, **extraparams)

        and return a numpy array of the same size as `mags_array` with the
        smoothed time-series. Any extra params can be provided using the
        `extraparams` dict.

    epdsmooth_extraparams : dict
        This is a dict of any extra filter params to supply to the smoothing
        function.

    nworkers : int
        The number of parallel workers to launch when processing the LCs.

    maxworkertasks : int
        The maximum number of tasks a parallel worker will complete before it is
        replaced with a new one (sometimes helps with memory-leaks).

    Returns
    -------

    dict
        Returns a dict organized by all the keys in the input `magcols` list,
        containing lists of EPD pickle light curves for that `magcol`.

    Notes
    -----

    - S -> measure of PSF sharpness (~1/sigma^2 sosmaller S = wider PSF)
    - D -> measure of PSF ellipticity in xy direction
    - K -> measure of PSF ellipticity in cross direction

    S, D, K are related to the PSF's variance and covariance, see eqn 30-33 in
    A. Pal's thesis: https://arxiv.org/abs/0906.3486


try:<EOL><INDENT>formatinfo = get_lcformat(lcformat,<EOL>use_lcformat_dir=lcformatdir)<EOL>if formatinfo:<EOL><INDENT>(fileglob, readerfunc,<EOL>dtimecols, dmagcols, derrcols,<EOL>magsarefluxes, normfunc) = formatinfo<EOL><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>")<EOL>return None<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION("<STR_LIT>")<EOL>return None<EOL><DEDENT>if lcfileglob is None:<EOL><INDENT>lcfileglob = fileglob<EOL><DEDENT>lclist = sorted(glob.glob(os.path.join(lcdir, lcfileglob)))<EOL>return parallel_epd_lclist(<EOL>lclist,<EOL>externalparams,<EOL>timecols=timecols,<EOL>magcols=magcols,<EOL>errcols=errcols,<EOL>lcformat=lcformat,<EOL>epdsmooth_sigclip=epdsmooth_sigclip,<EOL>epdsmooth_windowsize=epdsmooth_windowsize,<EOL>epdsmooth_func=epdsmooth_func,<EOL>epdsmooth_extraparams=epdsmooth_extraparams,<EOL>nworkers=nworkers,<EOL>maxworkertasks=maxworkertasks<EOL>)<EOL>

This applies EPD in parallel to all LCs in a directory.

    Parameters
    ----------

    lcdir : str
        The light curve directory to process.

    externalparams : dict or None
        This is a dict that indicates which keys in the lcdict obtained from the
        lcfile correspond to the required external parameters. As with timecol,
        magcol, and errcol, these can be simple keys (e.g. 'rjd') or compound
        keys ('magaperture1.mags'). The dict should look something like::

          {'fsv':'<lcdict key>' array: S values for each observation,
           'fdv':'<lcdict key>' array: D values for each observation,
           'fkv':'<lcdict key>' array: K values for each observation,
           'xcc':'<lcdict key>' array: x coords for each observation,
           'ycc':'<lcdict key>' array: y coords for each observation,
           'bgv':'<lcdict key>' array: sky background for each observation,
           'bge':'<lcdict key>' array: sky background err for each observation,
           'iha':'<lcdict key>' array: hour angle for each observation,
           'izd':'<lcdict key>' array: zenith distance for each observation}

    lcfileglob : str or None
        A UNIX fileglob to use to select light curve files in `lcdir`. If this
        is not None, the value provided will override the default fileglob for
        your light curve format.

    timecols,magcols,errcols : lists of str
        The keys in the lcdict produced by your light curve reader function that
        correspond to the times, mags/fluxes, and associated measurement errors
        that will be used as inputs to the EPD process. If these are None, the
        default values for `timecols`, `magcols`, and `errcols` for your light
        curve format will be used here.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    epdsmooth_sigclip : float or int or sequence of two floats/ints or None
        This specifies how to sigma-clip the input LC before fitting the EPD
        function to it.

        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    epdsmooth_windowsize : int
        This is the number of LC points to smooth over to generate a smoothed
        light curve that will be used to fit the EPD function.

    epdsmooth_func : Python function
        This sets the smoothing filter function to use. A Savitsky-Golay filter
        is used to smooth the light curve by default. The functions that can be
        used with this kwarg are listed in `varbase.trends`. If you want to use
        your own function, it MUST have the following signature::

                def smoothfunc(mags_array, window_size, **extraparams)

        and return a numpy array of the same size as `mags_array` with the
        smoothed time-series. Any extra params can be provided using the
        `extraparams` dict.

    epdsmooth_extraparams : dict
        This is a dict of any extra filter params to supply to the smoothing
        function.

    nworkers : int
        The number of parallel workers to launch when processing the LCs.

    maxworkertasks : int
        The maximum number of tasks a parallel worker will complete before it is
        replaced with a new one (sometimes helps with memory-leaks).

    Returns
    -------

    dict
        Returns a dict organized by all the keys in the input `magcols` list,
        containing lists of EPD pickle light curves for that `magcol`.

    Notes
    -----

    - S -> measure of PSF sharpness (~1/sigma^2 sosmaller S = wider PSF)
    - D -> measure of PSF ellipticity in xy direction
    - K -> measure of PSF ellipticity in cross direction

    S, D, K are related to the PSF's variance and covariance, see eqn 30-33 in
    A. Pal's thesis: https://arxiv.org/abs/0906.3486


try:<EOL><INDENT>formatinfo = get_lcformat(lcformat,<EOL>use_lcformat_dir=lcformatdir)<EOL>if formatinfo:<EOL><INDENT>(dfileglob, readerfunc,<EOL>dtimecols, dmagcols, derrcols,<EOL>magsarefluxes, normfunc) = formatinfo<EOL><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>")<EOL>return checkplotdict<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION("<STR_LIT>")<EOL>return checkplotdict<EOL><DEDENT>if not ('<STR_LIT>' in checkplotdict and<EOL>checkplotdict['<STR_LIT>'] and<EOL>len(checkplotdict['<STR_LIT>']) > <NUM_LIT:0>):<EOL><INDENT>LOGERROR('<STR_LIT>' %<EOL>(checkplotdict['<STR_LIT>']))<EOL>return checkplotdict<EOL><DEDENT>objmagkeys = {}<EOL>if '<STR_LIT>' in checkplotdict['<STR_LIT>']:<EOL><INDENT>mclist = checkplotdict['<STR_LIT>']['<STR_LIT>']<EOL><DEDENT>else:<EOL><INDENT>mclist = ('<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>',<EOL>'<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>')<EOL><DEDENT>for mc in mclist:<EOL><INDENT>if (mc in checkplotdict['<STR_LIT>'] and<EOL>checkplotdict['<STR_LIT>'][mc] is not None and<EOL>np.isfinite(checkplotdict['<STR_LIT>'][mc])):<EOL><INDENT>objmagkeys[mc] = checkplotdict['<STR_LIT>'][mc]<EOL><DEDENT><DEDENT>for nbr in checkplotdict['<STR_LIT>']:<EOL><INDENT>objectid, lcfpath = (nbr['<STR_LIT>'],<EOL>nbr['<STR_LIT>'])<EOL>if not os.path.exists(lcfpath):<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>' %<EOL>(checkplotdict['<STR_LIT>'], objectid, lcfpath))<EOL>continue<EOL><DEDENT>lcdict = readerfunc(lcfpath)<EOL>if ( (isinstance(lcdict, (list, tuple))) and<EOL>(isinstance(lcdict[<NUM_LIT:0>], dict)) ):<EOL><INDENT>lcdict = lcdict[<NUM_LIT:0>]<EOL><DEDENT>nbrmagkeys = {}<EOL>for mc in objmagkeys:<EOL><INDENT>if (('<STR_LIT>' in lcdict) and<EOL>(isinstance(lcdict['<STR_LIT>'], dict)) and<EOL>(mc in lcdict['<STR_LIT>']) and<EOL>(lcdict['<STR_LIT>'][mc] is not None) and<EOL>(np.isfinite(lcdict['<STR_LIT>'][mc]))):<EOL><INDENT>nbrmagkeys[mc] = lcdict['<STR_LIT>'][mc]<EOL><DEDENT><DEDENT>magdiffs = {}<EOL>for omc in objmagkeys:<EOL><INDENT>if omc in nbrmagkeys:<EOL><INDENT>magdiffs[omc] = objmagkeys[omc] - nbrmagkeys[omc]<EOL><DEDENT><DEDENT>colordiffs = {}<EOL>for ctrio in (['<STR_LIT>','<STR_LIT>','<STR_LIT>'],<EOL>['<STR_LIT>','<STR_LIT>','<STR_LIT>'],<EOL>['<STR_LIT>','<STR_LIT>','<STR_LIT>'],<EOL>['<STR_LIT>','<STR_LIT>','<STR_LIT>'],<EOL>['<STR_LIT>','<STR_LIT>','<STR_LIT>'],<EOL>['<STR_LIT>','<STR_LIT>','<STR_LIT>']):<EOL><INDENT>m1, m2, color = ctrio<EOL>if (m1 in objmagkeys and<EOL>m2 in objmagkeys and<EOL>m1 in nbrmagkeys and<EOL>m2 in nbrmagkeys):<EOL><INDENT>objcolor = objmagkeys[m1] - objmagkeys[m2]<EOL>nbrcolor = nbrmagkeys[m1] - nbrmagkeys[m2]<EOL>colordiffs[color] = objcolor - nbrcolor<EOL><DEDENT><DEDENT>nbr.update({'<STR_LIT>':magdiffs,<EOL>'<STR_LIT>':colordiffs})<EOL>if normfunc is not None:<EOL><INDENT>lcdict = normfunc(lcdict)<EOL><DEDENT>try:<EOL><INDENT>if '<STR_LIT:.>' in timecol:<EOL><INDENT>timecolget = timecol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>timecolget = [timecol]<EOL><DEDENT>times = _dict_get(lcdict, timecolget)<EOL>if '<STR_LIT:.>' in magcol:<EOL><INDENT>magcolget = magcol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>magcolget = [magcol]<EOL><DEDENT>mags = _dict_get(lcdict, magcolget)<EOL>if '<STR_LIT:.>' in errcol:<EOL><INDENT>errcolget = errcol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>errcolget = [errcol]<EOL><DEDENT>errs = _dict_get(lcdict, errcolget)<EOL><DEDENT>except KeyError:<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>' %<EOL>(objectid, checkplotdict['<STR_LIT>'],<EOL>'<STR_LIT:U+002CU+0020>'.join([timecol, magcol, errcol])))<EOL>continue<EOL><DEDENT>stimes, smags, serrs = sigclip_magseries(times,<EOL>mags,<EOL>errs,<EOL>magsarefluxes=magsarefluxes,<EOL>sigclip=<NUM_LIT>)<EOL>if normfunc is None:<EOL><INDENT>ntimes, nmags = normalize_magseries(<EOL>stimes, smags,<EOL>magsarefluxes=magsarefluxes<EOL>)<EOL>xtimes, xmags, xerrs = ntimes, nmags, serrs<EOL><DEDENT>else:<EOL><INDENT>xtimes, xmags, xerrs = stimes, smags, serrs<EOL><DEDENT>if ((xtimes is None) or (xmags is None) or (xerrs is None) or<EOL>(xtimes.size < <NUM_LIT>) or (xmags.size < <NUM_LIT>) or (xerrs.size < <NUM_LIT>)):<EOL><INDENT>LOGERROR("<STR_LIT>"<EOL>"<STR_LIT>"<EOL>"<STR_LIT>"<EOL>"<STR_LIT>" %<EOL>(checkplotdict['<STR_LIT>'],<EOL>nbr['<STR_LIT>'],<EOL>nbr['<STR_LIT>']))<EOL>continue<EOL><DEDENT>nbrdict = _pkl_magseries_plot(xtimes,<EOL>xmags,<EOL>xerrs,<EOL>magsarefluxes=magsarefluxes)<EOL>nbr.update(nbrdict)<EOL>if '<STR_LIT>' in checkplotdict:<EOL><INDENT>pfmethods = checkplotdict['<STR_LIT>']<EOL><DEDENT>else:<EOL><INDENT>pfmethods = []<EOL>for cpkey in checkplotdict:<EOL><INDENT>for pfkey in PFMETHODS:<EOL><INDENT>if pfkey in cpkey:<EOL><INDENT>pfmethods.append(pfkey)<EOL><DEDENT><DEDENT><DEDENT><DEDENT>for lspt in pfmethods:<EOL><INDENT>nbr[lspt] = {}<EOL>operiod, oepoch = (checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'],<EOL>checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'])<EOL>(ophasewrap, ophasesort, ophasebin,<EOL>ominbinelems, oplotxlim) = (<EOL>checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'],<EOL>checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'],<EOL>checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'],<EOL>checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'],<EOL>checkplotdict[lspt][<NUM_LIT:0>]['<STR_LIT>'],<EOL>)<EOL>nbr = _pkl_phased_magseries_plot(<EOL>nbr,<EOL>lspt.split('<STR_LIT:->')[<NUM_LIT:1>],  <EOL><NUM_LIT:0>,<EOL>xtimes, xmags, xerrs,<EOL>operiod, oepoch,<EOL>phasewrap=ophasewrap,<EOL>phasesort=ophasesort,<EOL>phasebin=ophasebin,<EOL>minbinelems=ominbinelems,<EOL>plotxlim=oplotxlim,<EOL>magsarefluxes=magsarefluxes,<EOL>verbose=verbose,<EOL>override_pfmethod=lspt<EOL>)<EOL><DEDENT><DEDENT>return checkplotdict<EOL>

For all neighbors in a checkplotdict, make LCs and phased LCs.

    Parameters
    ----------

    checkplotdict : dict
        This is the checkplot to process. The light curves for the neighbors to
        the object here will be extracted from the stored file paths, and this
        function will make plots of these time-series. If the object has 'best'
        periods and epochs generated by period-finder functions in this
        checkplotdict, phased light curve plots of each neighbor will be made
        using these to check the effects of blending.

    timecol,magcol,errcol : str
        The timecol, magcol, and errcol keys used to generate this object's
        checkplot. This is used to extract the correct times-series from the
        neighbors' light curves.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    Returns
    -------

    dict
        The input checkplotdict is returned with the neighor light curve plots
        added in.


try:<EOL><INDENT>formatinfo = get_lcformat(lcformat,<EOL>use_lcformat_dir=lcformatdir)<EOL>if formatinfo:<EOL><INDENT>(fileglob, readerfunc,<EOL>dtimecols, dmagcols, derrcols,<EOL>magsarefluxes, normfunc) = formatinfo<EOL><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>")<EOL>return None<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION("<STR_LIT>")<EOL>return None<EOL><DEDENT>if pfpickle is not None:<EOL><INDENT>if pfpickle.endswith('<STR_LIT>'):<EOL><INDENT>infd = gzip.open(pfpickle,'<STR_LIT:rb>')<EOL><DEDENT>else:<EOL><INDENT>infd = open(pfpickle,'<STR_LIT:rb>')<EOL><DEDENT>pfresults = pickle.load(infd)<EOL>infd.close()<EOL><DEDENT>if timecols is None:<EOL><INDENT>timecols = dtimecols<EOL><DEDENT>if magcols is None:<EOL><INDENT>magcols = dmagcols<EOL><DEDENT>if errcols is None:<EOL><INDENT>errcols = derrcols<EOL><DEDENT>if ((lcfname is not None or pfpickle is None) and os.path.exists(lcfname)):<EOL><INDENT>lcfpath = lcfname<EOL>objectid = None<EOL><DEDENT>else:<EOL><INDENT>if pfpickle is not None:<EOL><INDENT>objectid = pfresults['<STR_LIT>']<EOL>lcfbasename = pfresults['<STR_LIT>']<EOL>lcfsearchpath = os.path.join(lcbasedir, lcfbasename)<EOL>if os.path.exists(lcfsearchpath):<EOL><INDENT>lcfpath = lcfsearchpath<EOL><DEDENT>elif lcfname is not None and os.path.exists(lcfname):<EOL><INDENT>lcfpath = lcfname<EOL><DEDENT>else:<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>' %<EOL>(pfpickle, objectid, lcfsearchpath, lcfname))<EOL>return None<EOL><DEDENT><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>"<EOL>"<STR_LIT>")<EOL>return None<EOL><DEDENT><DEDENT>lcdict = readerfunc(lcfpath)<EOL>if ( (isinstance(lcdict, (list, tuple))) and<EOL>(isinstance(lcdict[<NUM_LIT:0>], dict)) ):<EOL><INDENT>lcdict = lcdict[<NUM_LIT:0>]<EOL><DEDENT>if objectid is None:<EOL><INDENT>if '<STR_LIT>' in lcdict:<EOL><INDENT>objectid = lcdict['<STR_LIT>']<EOL><DEDENT>elif ('<STR_LIT>' in lcdict['<STR_LIT>'] and<EOL>lcdict['<STR_LIT>']['<STR_LIT>']):<EOL><INDENT>objectid = lcdict['<STR_LIT>']['<STR_LIT>']<EOL><DEDENT>elif '<STR_LIT>' in lcdict['<STR_LIT>'] and lcdict['<STR_LIT>']['<STR_LIT>']:<EOL><INDENT>objectid = lcdict['<STR_LIT>']['<STR_LIT>']<EOL><DEDENT>else:<EOL><INDENT>objectid = uuid.uuid4().hex[:<NUM_LIT:5>]<EOL>LOGWARNING('<STR_LIT>'<EOL>'<STR_LIT>' % objectid)<EOL><DEDENT><DEDENT>if normfunc is not None:<EOL><INDENT>lcdict = normfunc(lcdict)<EOL><DEDENT>cpfs = []<EOL>for tcol, mcol, ecol in zip(timecols, magcols, errcols):<EOL><INDENT>if '<STR_LIT:.>' in tcol:<EOL><INDENT>tcolget = tcol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>tcolget = [tcol]<EOL><DEDENT>times = _dict_get(lcdict, tcolget)<EOL>if '<STR_LIT:.>' in mcol:<EOL><INDENT>mcolget = mcol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>mcolget = [mcol]<EOL><DEDENT>mags = _dict_get(lcdict, mcolget)<EOL>if '<STR_LIT:.>' in ecol:<EOL><INDENT>ecolget = ecol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>ecolget = [ecol]<EOL><DEDENT>errs = _dict_get(lcdict, ecolget)<EOL>if pfpickle is not None:<EOL><INDENT>if '<STR_LIT>' in pfresults[mcol]:<EOL><INDENT>pflist = [<EOL>pfresults[mcol][x] for x in<EOL>pfresults[mcol]['<STR_LIT>'] if<EOL>len(pfresults[mcol][x].keys()) > <NUM_LIT:0><EOL>]<EOL><DEDENT>else:<EOL><INDENT>pflist = []<EOL>for pfm in PFMETHODS:<EOL><INDENT>if (pfm in pfresults[mcol] and<EOL>len(pfresults[mcol][pfm].keys()) > <NUM_LIT:0>):<EOL><INDENT>pflist.append(pfresults[mcol][pfm])<EOL><DEDENT><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>pflist = []<EOL><DEDENT>outfile = os.path.join(outdir,<EOL>'<STR_LIT>' % (<EOL>squeeze(objectid).replace('<STR_LIT:U+0020>','<STR_LIT:->'),<EOL>mcol<EOL>))<EOL>if skipdone and os.path.exists(outfile):<EOL><INDENT>LOGWARNING('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>' % outfile)<EOL>return outfile<EOL><DEDENT>if '<STR_LIT>' not in lcdict['<STR_LIT>']:<EOL><INDENT>lcdict['<STR_LIT>']['<STR_LIT>'] = objectid<EOL><DEDENT>if normfunc is None:<EOL><INDENT>ntimes, nmags = normalize_magseries(<EOL>times, mags,<EOL>magsarefluxes=magsarefluxes<EOL>)<EOL>xtimes, xmags, xerrs = ntimes, nmags, errs<EOL><DEDENT>else:<EOL><INDENT>xtimes, xmags, xerrs = times, mags, errs<EOL><DEDENT>cpd = checkplot_dict(<EOL>pflist,<EOL>xtimes, xmags, xerrs,<EOL>objectinfo=lcdict['<STR_LIT>'],<EOL>gaia_max_timeout=gaia_max_timeout,<EOL>gaia_mirror=gaia_mirror,<EOL>lclistpkl=lclistpkl,<EOL>nbrradiusarcsec=nbrradiusarcsec,<EOL>maxnumneighbors=maxnumneighbors,<EOL>xmatchinfo=xmatchinfo,<EOL>xmatchradiusarcsec=xmatchradiusarcsec,<EOL>sigclip=sigclip,<EOL>mindet=minobservations,<EOL>verbose=False,<EOL>fast_mode=fast_mode,<EOL>magsarefluxes=magsarefluxes,<EOL>normto=cprenorm  <EOL>)<EOL>if makeneighborlcs:<EOL><INDENT>cpdupdated = update_checkplotdict_nbrlcs(<EOL>cpd,<EOL>tcol, mcol, ecol,<EOL>lcformat=lcformat,<EOL>verbose=False<EOL>)<EOL><DEDENT>else:<EOL><INDENT>cpdupdated = cpd<EOL><DEDENT>cpf = _write_checkplot_picklefile(<EOL>cpdupdated,<EOL>outfile=outfile,<EOL>protocol=pickle.HIGHEST_PROTOCOL,<EOL>outgzip=False<EOL>)<EOL>cpfs.append(cpf)<EOL><DEDENT>LOGINFO('<STR_LIT>' % (objectid, repr(cpfs)))<EOL>if done_callback is not None:<EOL><INDENT>if (done_callback_args is not None and<EOL>isinstance(done_callback_args,list)):<EOL><INDENT>done_callback_args = tuple([cpfs] + done_callback_args)<EOL><DEDENT>else:<EOL><INDENT>done_callback_args = (cpfs,)<EOL><DEDENT>if (done_callback_kwargs is not None and<EOL>isinstance(done_callback_kwargs, dict)):<EOL><INDENT>done_callback_kwargs.update(dict(<EOL>fast_mode=fast_mode,<EOL>lcfname=lcfname,<EOL>cprenorm=cprenorm,<EOL>lclistpkl=lclistpkl,<EOL>nbrradiusarcsec=nbrradiusarcsec,<EOL>maxnumneighbors=maxnumneighbors,<EOL>gaia_max_timeout=gaia_max_timeout,<EOL>gaia_mirror=gaia_mirror,<EOL>xmatchinfo=xmatchinfo,<EOL>xmatchradiusarcsec=xmatchradiusarcsec,<EOL>minobservations=minobservations,<EOL>sigclip=sigclip,<EOL>lcformat=lcformat,<EOL>fileglob=fileglob,<EOL>readerfunc=readerfunc,<EOL>normfunc=normfunc,<EOL>magsarefluxes=magsarefluxes,<EOL>timecols=timecols,<EOL>magcols=magcols,<EOL>errcols=errcols,<EOL>skipdone=skipdone,<EOL>))<EOL><DEDENT>else:<EOL><INDENT>done_callback_kwargs = dict(<EOL>fast_mode=fast_mode,<EOL>lcfname=lcfname,<EOL>cprenorm=cprenorm,<EOL>lclistpkl=lclistpkl,<EOL>nbrradiusarcsec=nbrradiusarcsec,<EOL>maxnumneighbors=maxnumneighbors,<EOL>gaia_max_timeout=gaia_max_timeout,<EOL>gaia_mirror=gaia_mirror,<EOL>xmatchinfo=xmatchinfo,<EOL>xmatchradiusarcsec=xmatchradiusarcsec,<EOL>minobservations=minobservations,<EOL>sigclip=sigclip,<EOL>lcformat=lcformat,<EOL>fileglob=fileglob,<EOL>readerfunc=readerfunc,<EOL>normfunc=normfunc,<EOL>magsarefluxes=magsarefluxes,<EOL>timecols=timecols,<EOL>magcols=magcols,<EOL>errcols=errcols,<EOL>skipdone=skipdone,<EOL>)<EOL><DEDENT>try:<EOL><INDENT>done_callback(*done_callback_args, **done_callback_kwargs)<EOL>LOGINFO('<STR_LIT>' % cpfs)<EOL><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>' % cpfs)<EOL><DEDENT><DEDENT>return cpfs<EOL>

This makes a checkplot pickle for the given period-finding result pickle
    produced by `lcproc.periodfinding.runpf`.

    Parameters
    ----------

    pfpickle : str or None
        This is the filename of the period-finding result pickle file created by
        `lcproc.periodfinding.runpf`. If this is None, the checkplot will be
        made anyway, but no phased LC information will be collected into the
        output checkplot pickle. This can be useful for just collecting GAIA and
        other external information and making LC plots for an object.

    outdir : str
        This is the directory to which the output checkplot pickle will be
        written.

    lcbasedir : str
        The base directory where this function will look for the light curve
        file associated with the object in the input period-finding result
        pickle file.

    lcfname : str or None
        This is usually None because we'll get the path to the light curve
        associated with this period-finding pickle from the pickle itself. If
        `pfpickle` is None, however, this function will use `lcfname` to look up
        the light curve file instead. If both are provided, the value of
        `lcfname` takes precedence.

        Providing the light curve file name in this kwarg is useful when you're
        making checkplots directly from light curve files and not including
        period-finder results (perhaps because period-finding takes a long time
        for large collections of LCs).

    cprenorm : bool
        Set this to True if the light curves should be renormalized by
        `checkplot.checkplot_pickle`. This is set to False by default because we
        do our own normalization in this function using the light curve's
        registered normalization function and pass the normalized times, mags,
        errs to the `checkplot.checkplot_pickle` function.

    lclistpkl : str or dict
        This is either the filename of a pickle or the actual dict produced by
        lcproc.make_lclist. This is used to gather neighbor information.

    nbrradiusarcsec : float
        The radius in arcseconds to use for a search conducted around the
        coordinates of this object to look for any potential confusion and
        blending of variability amplitude caused by their proximity.

    maxnumneighbors : int
        The maximum number of neighbors that will have their light curves and
        magnitudes noted in this checkplot as potential blends with the target
        object.

    makeneighborlcs : bool
        If True, will make light curve and phased light curve plots for all
        neighbors to the current object found in the catalog passed in using
        `lclistpkl`.

    fast_mode : bool or float
        This runs the external catalog operations in a "fast" mode, with short
        timeouts and not trying to hit external catalogs that take a long time
        to respond.

        If this is set to True, the default settings for the external requests
        will then become::

                skyview_lookup = False
                skyview_timeout = 10.0
                skyview_retry_failed = False
                dust_timeout = 10.0
                gaia_submit_timeout = 7.0
                gaia_max_timeout = 10.0
                gaia_submit_tries = 2
                complete_query_later = False
                search_simbad = False

        If this is a float, will run in "fast" mode with the provided timeout
        value in seconds and the following settings::

                skyview_lookup = True
                skyview_timeout = fast_mode
                skyview_retry_failed = False
                dust_timeout = fast_mode
                gaia_submit_timeout = 0.66*fast_mode
                gaia_max_timeout = fast_mode
                gaia_submit_tries = 2
                complete_query_later = False
                search_simbad = False

    gaia_max_timeout : float
        Sets the timeout in seconds to use when waiting for the GAIA service to
        respond to our request for the object's information. Note that if
        `fast_mode` is set, this is ignored.

    gaia_mirror : str or None
        This sets the GAIA mirror to use. This is a key in the
        `services.gaia.GAIA_URLS` dict which defines the URLs to hit for each
        mirror.

    xmatchinfo : str or dict
        This is either the xmatch dict produced by the function
        `load_xmatch_external_catalogs` above, or the path to the xmatch info
        pickle file produced by that function.

    xmatchradiusarcsec : float
        This is the cross-matching radius to use in arcseconds.

    minobservations : int
        The minimum of observations the input object's mag/flux time-series must
        have for this function to plot its light curve and phased light
        curve. If the object has less than this number, no light curves will be
        plotted, but the checkplotdict will still contain all of the other
        information.

    sigclip : float or int or sequence of two floats/ints or None
        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    timecols : list of str or None
        The timecol keys to use from the lcdict in generating this checkplot.

    magcols : list of str or None
        The magcol keys to use from the lcdict in generating this checkplot.

    errcols : list of str or None
        The errcol keys to use from the lcdict in generating this checkplot.

    skipdone : bool
        This indicates if this function will skip creating checkplots that
        already exist corresponding to the current `objectid` and `magcol`. If
        `skipdone` is set to True, this will be done.

    done_callback : Python function or None
        This is used to provide a function to execute after the checkplot
        pickles are generated. This is useful if you want to stream the results
        of checkplot making to some other process, e.g. directly running an
        ingestion into an LCC-Server collection. The function will always get
        the list of the generated checkplot pickles as its first arg, and all of
        the kwargs for runcp in the kwargs dict. Additional args and kwargs can
        be provided by giving a list in the `done_callbacks_args` kwarg and a
        dict in the `done_callbacks_kwargs` kwarg.

        NOTE: the function you pass in here should be pickleable by normal
        Python if you want to use it with the parallel_cp and parallel_cp_lcdir
        functions below.

    done_callback_args : tuple or None
        If not None, contains any args to pass into the `done_callback`
        function.

    done_callback_kwargs : dict or None
        If not None, contains any kwargs to pass into the `done_callback`
        function.

    Returns
    -------

    list of str
        This returns a list of checkplot pickle filenames with one element for
        each (timecol, magcol, errcol) combination provided in the default
        lcformat config or in the timecols, magcols, errcols kwargs.


pfpickle, outdir, lcbasedir, kwargs = task<EOL>try:<EOL><INDENT>return runcp(pfpickle, outdir, lcbasedir, **kwargs)<EOL><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>' % (pfpickle, e))<EOL>return None<EOL><DEDENT>

This is the worker for running checkplots.

Parameters
----------

task : tuple
    This is of the form: (pfpickle, outdir, lcbasedir, kwargs).

Returns
-------

list of str
    The list of checkplot pickles returned by the `runcp` function.


<EOL>if sys.platform == '<STR_LIT>':<EOL><INDENT>import requests<EOL>requests.get('<STR_LIT>')<EOL><DEDENT>if not os.path.exists(outdir):<EOL><INDENT>os.mkdir(outdir)<EOL><DEDENT>if (liststartindex is not None) and (maxobjects is None):<EOL><INDENT>pfpicklelist = pfpicklelist[liststartindex:]<EOL>if lcfnamelist is not None:<EOL><INDENT>lcfnamelist = lcfnamelist[liststartindex:]<EOL><DEDENT><DEDENT>elif (liststartindex is None) and (maxobjects is not None):<EOL><INDENT>pfpicklelist = pfpicklelist[:maxobjects]<EOL>if lcfnamelist is not None:<EOL><INDENT>lcfnamelist = lcfnamelist[:maxobjects]<EOL><DEDENT><DEDENT>elif (liststartindex is not None) and (maxobjects is not None):<EOL><INDENT>pfpicklelist = (<EOL>pfpicklelist[liststartindex:liststartindex+maxobjects]<EOL>)<EOL>if lcfnamelist is not None:<EOL><INDENT>lcfnamelist = lcfnamelist[liststartindex:liststartindex+maxobjects]<EOL><DEDENT><DEDENT>if lcfnamelist is None:<EOL><INDENT>lcfnamelist = [None]*len(pfpicklelist)<EOL><DEDENT>tasklist = [(x, outdir, lcbasedir,<EOL>{'<STR_LIT>':lcformat,<EOL>'<STR_LIT>':lcformatdir,<EOL>'<STR_LIT>':y,<EOL>'<STR_LIT>':timecols,<EOL>'<STR_LIT>':magcols,<EOL>'<STR_LIT>':errcols,<EOL>'<STR_LIT>':lclistpkl,<EOL>'<STR_LIT>':gaia_max_timeout,<EOL>'<STR_LIT>':gaia_mirror,<EOL>'<STR_LIT>':nbrradiusarcsec,<EOL>'<STR_LIT>':maxnumneighbors,<EOL>'<STR_LIT>':makeneighborlcs,<EOL>'<STR_LIT>':xmatchinfo,<EOL>'<STR_LIT>':xmatchradiusarcsec,<EOL>'<STR_LIT>':sigclip,<EOL>'<STR_LIT>':minobservations,<EOL>'<STR_LIT>':skipdone,<EOL>'<STR_LIT>':cprenorm,<EOL>'<STR_LIT>':fast_mode,<EOL>'<STR_LIT>':done_callback,<EOL>'<STR_LIT>':done_callback_args,<EOL>'<STR_LIT>':done_callback_kwargs}) for<EOL>x,y in zip(pfpicklelist, lcfnamelist)]<EOL>resultfutures = []<EOL>results = []<EOL>with ProcessPoolExecutor(max_workers=nworkers) as executor:<EOL><INDENT>resultfutures = executor.map(runcp_worker, tasklist)<EOL><DEDENT>results = [x for x in resultfutures]<EOL>executor.shutdown()<EOL>return results<EOL>

This drives the parallel execution of `runcp` for a list of periodfinding
    result pickles.

    Parameters
    ----------

    pfpicklelist : list of str or list of Nones
        This is the list of the filenames of the period-finding result pickles
        to process. To make checkplots using the light curves directly, set this
        to a list of Nones with the same length as the list of light curve files
        that you provide in `lcfnamelist`.

    outdir : str
        The directory the checkplot pickles will be written to.

    lcbasedir : str
        The base directory that this function will look in to find the light
        curves pointed to by the period-finding result files. If you're using
        `lcfnamelist` to provide a list of light curve filenames directly, this
        arg is ignored.

    lcfnamelist : list of str or None
        If this is provided, it must be a list of the input light curve
        filenames to process. These can either be associated with each input
        period-finder result pickle, or can be provided standalone to make
        checkplots without phased LC plots in them. In the second case, you must
        set `pfpicklelist` to a list of Nones that matches the length of
        `lcfnamelist`.

    cprenorm : bool
        Set this to True if the light curves should be renormalized by
        `checkplot.checkplot_pickle`. This is set to False by default because we
        do our own normalization in this function using the light curve's
        registered normalization function and pass the normalized times, mags,
        errs to the `checkplot.checkplot_pickle` function.

    lclistpkl : str or dict
        This is either the filename of a pickle or the actual dict produced by
        lcproc.make_lclist. This is used to gather neighbor information.

    nbrradiusarcsec : float
        The radius in arcseconds to use for a search conducted around the
        coordinates of this object to look for any potential confusion and
        blending of variability amplitude caused by their proximity.

    maxnumneighbors : int
        The maximum number of neighbors that will have their light curves and
        magnitudes noted in this checkplot as potential blends with the target
        object.

    makeneighborlcs : bool
        If True, will make light curve and phased light curve plots for all
        neighbors found in the object collection for each input object.

    fast_mode : bool or float
        This runs the external catalog operations in a "fast" mode, with short
        timeouts and not trying to hit external catalogs that take a long time
        to respond.

        If this is set to True, the default settings for the external requests
        will then become::

                skyview_lookup = False
                skyview_timeout = 10.0
                skyview_retry_failed = False
                dust_timeout = 10.0
                gaia_submit_timeout = 7.0
                gaia_max_timeout = 10.0
                gaia_submit_tries = 2
                complete_query_later = False
                search_simbad = False

        If this is a float, will run in "fast" mode with the provided timeout
        value in seconds and the following settings::

                skyview_lookup = True
                skyview_timeout = fast_mode
                skyview_retry_failed = False
                dust_timeout = fast_mode
                gaia_submit_timeout = 0.66*fast_mode
                gaia_max_timeout = fast_mode
                gaia_submit_tries = 2
                complete_query_later = False
                search_simbad = False

    gaia_max_timeout : float
        Sets the timeout in seconds to use when waiting for the GAIA service to
        respond to our request for the object's information. Note that if
        `fast_mode` is set, this is ignored.

    gaia_mirror : str or None
        This sets the GAIA mirror to use. This is a key in the
        `services.gaia.GAIA_URLS` dict which defines the URLs to hit for each
        mirror.

    xmatchinfo : str or dict
        This is either the xmatch dict produced by the function
        `load_xmatch_external_catalogs` above, or the path to the xmatch info
        pickle file produced by that function.

    xmatchradiusarcsec : float
        This is the cross-matching radius to use in arcseconds.

    minobservations : int
        The minimum of observations the input object's mag/flux time-series must
        have for this function to plot its light curve and phased light
        curve. If the object has less than this number, no light curves will be
        plotted, but the checkplotdict will still contain all of the other
        information.

    sigclip : float or int or sequence of two floats/ints or None
        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    timecols : list of str or None
        The timecol keys to use from the lcdict in generating this checkplot.

    magcols : list of str or None
        The magcol keys to use from the lcdict in generating this checkplot.

    errcols : list of str or None
        The errcol keys to use from the lcdict in generating this checkplot.

    skipdone : bool
        This indicates if this function will skip creating checkplots that
        already exist corresponding to the current `objectid` and `magcol`. If
        `skipdone` is set to True, this will be done.

    done_callback : Python function or None
        This is used to provide a function to execute after the checkplot
        pickles are generated. This is useful if you want to stream the results
        of checkplot making to some other process, e.g. directly running an
        ingestion into an LCC-Server collection. The function will always get
        the list of the generated checkplot pickles as its first arg, and all of
        the kwargs for runcp in the kwargs dict. Additional args and kwargs can
        be provided by giving a list in the `done_callbacks_args` kwarg and a
        dict in the `done_callbacks_kwargs` kwarg.

        NOTE: the function you pass in here should be pickleable by normal
        Python if you want to use it with the parallel_cp and parallel_cp_lcdir
        functions below.

    done_callback_args : tuple or None
        If not None, contains any args to pass into the `done_callback`
        function.

    done_callback_kwargs : dict or None
        If not None, contains any kwargs to pass into the `done_callback`
        function.

    liststartindex : int
        The index of the `pfpicklelist` (and `lcfnamelist` if provided) to start
        working at.

    maxobjects : int
        The maximum number of objects to process in this run. Use this with
        `liststartindex` to effectively distribute working on a large list of
        input period-finding result pickles (and light curves if `lcfnamelist`
        is also provided) over several sessions or machines.

    nworkers : int
        The number of parallel workers that will work on the checkplot
        generation process.

    Returns
    -------

    dict
        This returns a dict with keys = input period-finding pickles and vals =
        list of the corresponding checkplot pickles produced.


This drives the parallel execution of `runcp` for a directory of
    periodfinding pickles.

    Parameters
    ----------

    pfpickledir : str
        This is the directory containing all of the period-finding pickles to
        process.

    outdir : str
        The directory the checkplot pickles will be written to.

    lcbasedir : str
        The base directory that this function will look in to find the light
        curves pointed to by the period-finding result files. If you're using
        `lcfnamelist` to provide a list of light curve filenames directly, this
        arg is ignored.

    pkpickleglob : str
        This is a UNIX file glob to select period-finding result pickles in the
        specified `pfpickledir`.

    lclistpkl : str or dict
        This is either the filename of a pickle or the actual dict produced by
        lcproc.make_lclist. This is used to gather neighbor information.

    cprenorm : bool
        Set this to True if the light curves should be renormalized by
        `checkplot.checkplot_pickle`. This is set to False by default because we
        do our own normalization in this function using the light curve's
        registered normalization function and pass the normalized times, mags,
        errs to the `checkplot.checkplot_pickle` function.

    nbrradiusarcsec : float
        The radius in arcseconds to use for a search conducted around the
        coordinates of this object to look for any potential confusion and
        blending of variability amplitude caused by their proximity.

    maxnumneighbors : int
        The maximum number of neighbors that will have their light curves and
        magnitudes noted in this checkplot as potential blends with the target
        object.

    makeneighborlcs : bool
        If True, will make light curve and phased light curve plots for all
        neighbors found in the object collection for each input object.

    fast_mode : bool or float
        This runs the external catalog operations in a "fast" mode, with short
        timeouts and not trying to hit external catalogs that take a long time
        to respond.

        If this is set to True, the default settings for the external requests
        will then become::

                skyview_lookup = False
                skyview_timeout = 10.0
                skyview_retry_failed = False
                dust_timeout = 10.0
                gaia_submit_timeout = 7.0
                gaia_max_timeout = 10.0
                gaia_submit_tries = 2
                complete_query_later = False
                search_simbad = False

        If this is a float, will run in "fast" mode with the provided timeout
        value in seconds and the following settings::

                skyview_lookup = True
                skyview_timeout = fast_mode
                skyview_retry_failed = False
                dust_timeout = fast_mode
                gaia_submit_timeout = 0.66*fast_mode
                gaia_max_timeout = fast_mode
                gaia_submit_tries = 2
                complete_query_later = False
                search_simbad = False

    gaia_max_timeout : float
        Sets the timeout in seconds to use when waiting for the GAIA service to
        respond to our request for the object's information. Note that if
        `fast_mode` is set, this is ignored.

    gaia_mirror : str or None
        This sets the GAIA mirror to use. This is a key in the
        `services.gaia.GAIA_URLS` dict which defines the URLs to hit for each
        mirror.

    xmatchinfo : str or dict
        This is either the xmatch dict produced by the function
        `load_xmatch_external_catalogs` above, or the path to the xmatch info
        pickle file produced by that function.

    xmatchradiusarcsec : float
        This is the cross-matching radius to use in arcseconds.

    minobservations : int
        The minimum of observations the input object's mag/flux time-series must
        have for this function to plot its light curve and phased light
        curve. If the object has less than this number, no light curves will be
        plotted, but the checkplotdict will still contain all of the other
        information.

    sigclip : float or int or sequence of two floats/ints or None
        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    timecols : list of str or None
        The timecol keys to use from the lcdict in generating this checkplot.

    magcols : list of str or None
        The magcol keys to use from the lcdict in generating this checkplot.

    errcols : list of str or None
        The errcol keys to use from the lcdict in generating this checkplot.

    skipdone : bool
        This indicates if this function will skip creating checkplots that
        already exist corresponding to the current `objectid` and `magcol`. If
        `skipdone` is set to True, this will be done.

    done_callback : Python function or None
        This is used to provide a function to execute after the checkplot
        pickles are generated. This is useful if you want to stream the results
        of checkplot making to some other process, e.g. directly running an
        ingestion into an LCC-Server collection. The function will always get
        the list of the generated checkplot pickles as its first arg, and all of
        the kwargs for runcp in the kwargs dict. Additional args and kwargs can
        be provided by giving a list in the `done_callbacks_args` kwarg and a
        dict in the `done_callbacks_kwargs` kwarg.

        NOTE: the function you pass in here should be pickleable by normal
        Python if you want to use it with the parallel_cp and parallel_cp_lcdir
        functions below.

    done_callback_args : tuple or None
        If not None, contains any args to pass into the `done_callback`
        function.

    done_callback_kwargs : dict or None
        If not None, contains any kwargs to pass into the `done_callback`
        function.

    maxobjects : int
        The maximum number of objects to process in this run.

    nworkers : int
        The number of parallel workers that will work on the checkplot
        generation process.

    Returns
    -------

    dict
        This returns a dict with keys = input period-finding pickles and vals =
        list of the corresponding checkplot pickles produced.


if lcfile.endswith('<STR_LIT>'):<EOL><INDENT>try:<EOL><INDENT>with gzip.open(lcfile,'<STR_LIT:rb>') as infd:<EOL><INDENT>lcdict = pickle.load(infd)<EOL><DEDENT><DEDENT>except UnicodeDecodeError:<EOL><INDENT>with gzip.open(lcfile,'<STR_LIT:rb>') as infd:<EOL><INDENT>lcdict = pickle.load(infd, encoding='<STR_LIT>')<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>try:<EOL><INDENT>with open(lcfile,'<STR_LIT:rb>') as infd:<EOL><INDENT>lcdict = pickle.load(infd)<EOL><DEDENT><DEDENT>except UnicodeDecodeError:<EOL><INDENT>with open(lcfile,'<STR_LIT:rb>') as infd:<EOL><INDENT>lcdict = pickle.load(infd, encoding='<STR_LIT>')<EOL><DEDENT><DEDENT><DEDENT>return lcdict<EOL>

This just reads a light curve pickle file.

Parameters
----------

lcfile : str
    The file name of the pickle to open.

Returns
-------

dict
    This returns an lcdict.


try:<EOL><INDENT>if os.path.exists(module):<EOL><INDENT>sys.path.append(os.path.dirname(module))<EOL>importedok = importlib.import_module(<EOL>os.path.basename(module.replace('<STR_LIT>','<STR_LIT>'))<EOL>)<EOL><DEDENT>else:<EOL><INDENT>importedok = importlib.import_module(module)<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (module, formatkey))<EOL>importedok = False<EOL><DEDENT>return importedok<EOL>

This imports the module specified.

    Used to dynamically import Python modules that are needed to support LC
    formats not natively supported by astrobase.

    Parameters
    ----------

    module : str
        This is either:

        - a Python module import path, e.g. 'astrobase.lcproc.catalogs' or
        - a path to a Python file, e.g. '/astrobase/hatsurveys/hatlc.py'

        that contains the Python module that contains functions used to open
        (and optionally normalize) a custom LC format that's not natively
        supported by astrobase.

    formatkey : str
        A str used as the unique ID of this LC format for all lcproc functions
        and can be used to look it up later and import the correct functions
        needed to support it for lcproc operations. For example, we use
        'kep-fits' as a the specifier for Kepler FITS light curves, which can be
        read by the `astrobase.astrokep.read_kepler_fitslc` function as
        specified by the `<astrobase install path>/data/lcformats/kep-fits.json`
        LC format specification JSON.

    Returns
    -------

    Python module
        This returns a Python module if it's able to successfully import it.


LOGINFO('<STR_LIT>' % formatkey)<EOL>lcformat_dpath = os.path.abspath(<EOL>os.path.expanduser(lcformat_dir)<EOL>)<EOL>if not os.path.exists(lcformat_dpath):<EOL><INDENT>os.makedirs(lcformat_dpath)<EOL><DEDENT>lcformat_jsonpath = os.path.join(lcformat_dpath,'<STR_LIT>' % formatkey)<EOL>if os.path.exists(lcformat_jsonpath) and not overwrite_existing:<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (lcformat_jsonpath, formatkey))<EOL>return None<EOL><DEDENT>readermodule = _check_extmodule(readerfunc_module, formatkey)<EOL>if not readermodule:<EOL><INDENT>LOGERROR("<STR_LIT>"<EOL>"<STR_LIT>" %<EOL>(readerfunc_module, formatkey))<EOL>return None<EOL><DEDENT>try:<EOL><INDENT>getattr(readermodule, readerfunc)<EOL>readerfunc_in = readerfunc<EOL><DEDENT>except AttributeError:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (formatkey, readerfunc_module, readerfunc))<EOL>raise<EOL><DEDENT>if normfunc_module:<EOL><INDENT>normmodule = _check_extmodule(normfunc_module, formatkey)<EOL>if not normmodule:<EOL><INDENT>LOGERROR("<STR_LIT>"<EOL>"<STR_LIT>" %<EOL>(normfunc_module, formatkey))<EOL>return None<EOL><DEDENT><DEDENT>else:<EOL><INDENT>normmodule = None<EOL><DEDENT>if normfunc_module and normfunc:<EOL><INDENT>try:<EOL><INDENT>getattr(normmodule, normfunc)<EOL>normfunc_in = normfunc<EOL><DEDENT>except AttributeError:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (normfunc, formatkey, normfunc_module))<EOL>raise<EOL><DEDENT><DEDENT>else:<EOL><INDENT>normfunc_in = None<EOL><DEDENT>formatdict = {'<STR_LIT>':fileglob,<EOL>'<STR_LIT>':timecols,<EOL>'<STR_LIT>':magcols,<EOL>'<STR_LIT>':errcols,<EOL>'<STR_LIT>':magsarefluxes,<EOL>'<STR_LIT>':readerfunc_module,<EOL>'<STR_LIT>':readerfunc_in,<EOL>'<STR_LIT>':readerfunc_kwargs,<EOL>'<STR_LIT>':normfunc_module,<EOL>'<STR_LIT>':normfunc_in,<EOL>'<STR_LIT>':normfunc_kwargs}<EOL>with open(lcformat_jsonpath,'<STR_LIT:w>') as outfd:<EOL><INDENT>json.dump(formatdict, outfd, indent=<NUM_LIT:4>)<EOL><DEDENT>return lcformat_jsonpath<EOL>

This adds a new LC format to the astrobase LC format registry.

    Allows handling of custom format light curves for astrobase lcproc
    drivers. Once the format is successfully registered, light curves should
    work transparently with all of the functions in this module, by simply
    calling them with the `formatkey` in the `lcformat` keyword argument.

    LC format specifications are generated as JSON files. astrobase comes with
    several of these in `<astrobase install path>/data/lcformats`. LC formats
    you add by using this function will have their specifiers written to the
    `~/.astrobase/lcformat-jsons` directory in your home directory.

    Parameters
    ----------

    formatkey : str
        A str used as the unique ID of this LC format for all lcproc functions
        and can be used to look it up later and import the correct functions
        needed to support it for lcproc operations. For example, we use
        'kep-fits' as a the specifier for Kepler FITS light curves, which can be
        read by the `astrobase.astrokep.read_kepler_fitslc` function as
        specified by the `<astrobase install path>/data/lcformats/kep-fits.json`
        LC format specification JSON produced by `register_lcformat`.

    fileglob : str
        The default UNIX fileglob to use to search for light curve files in this
        LC format. This is a string like '*-whatever-???-*.*??-.lc'.

    timecols,magcols,errcols : list of str
        These are all lists of strings indicating which keys in the lcdict
        produced by your `lcreader_func` that will be extracted and used by
        lcproc functions for processing. The lists must all have the same
        dimensions, e.g. if timecols = ['timecol1','timecol2'], then magcols
        must be something like ['magcol1','magcol2'] and errcols must be
        something like ['errcol1', 'errcol2']. This allows you to process
        multiple apertures or multiple types of measurements in one go.

        Each element in these lists can be a simple key, e.g. 'time' (which
        would correspond to lcdict['time']), or a composite key,
        e.g. 'aperture1.times.rjd' (which would correspond to
        lcdict['aperture1']['times']['rjd']). See the examples in the lcformat
        specification JSON files in `<astrobase install path>/data/lcformats`.

    readerfunc_module : str
        This is either:

        - a Python module import path, e.g. 'astrobase.lcproc.catalogs' or
        - a path to a Python file, e.g. '/astrobase/hatsurveys/hatlc.py'

        that contains the Python module that contains functions used to open
        (and optionally normalize) a custom LC format that's not natively
        supported by astrobase.

    readerfunc : str
        This is the function name in `readerfunc_module` to use to read light
        curves in the custom format. This MUST always return a dictionary (the
        'lcdict') with the following signature (the keys listed below are
        required, but others are allowed)::

            {'objectid': this object's identifier as a string,
             'objectinfo':{'ra': this object's right ascension in decimal deg,
                           'decl': this object's declination in decimal deg,
                           'ndet': the number of observations in this LC,
                           'objectid': the object ID again for legacy reasons},
             ...other time columns, mag columns go in as their own keys}

    normfunc_kwargs : dict or None
        This is a dictionary containing any kwargs to pass through to
        the light curve norm function.

    normfunc_module : str or None
        This is either:

        - a Python module import path, e.g. 'astrobase.lcproc.catalogs' or
        - a path to a Python file, e.g. '/astrobase/hatsurveys/hatlc.py'
        - None, in which case we'll use default normalization

        that contains the Python module that contains functions used to
        normalize a custom LC format that's not natively supported by astrobase.

    normfunc : str or None
        This is the function name in `normfunc_module` to use to normalize light
        curves in the custom format. If None, the default normalization method
        used by lcproc is to find gaps in the time-series, normalize
        measurements grouped by these gaps to zero, then normalize the entire
        magnitude time series to global time series median using the
        `astrobase.lcmath.normalize_magseries` function.

        If this is provided, the normalization function should take and return
        an lcdict of the same form as that produced by `readerfunc` above. For
        an example of a specific normalization function, see
        `normalize_lcdict_by_inst` in the `astrobase.hatsurveys.hatlc` module.

    normfunc_kwargs : dict or None
        This is a dictionary containing any kwargs to pass through to
        the light curve normalization function.

    magsarefluxes : bool
        If this is True, then all lcproc functions will treat the measurement
        columns in the lcdict produced by your `readerfunc` as flux instead of
        mags, so things like default normalization and sigma-clipping will be
        done correctly. If this is False, magnitudes will be treated as
        magnitudes.

    overwrite_existing : bool
        If this is True, this function will overwrite any existing LC format
        specification JSON with the same name as that provided in the
        `formatkey` arg. This can be used to update LC format specifications
        while keeping the `formatkey` the same.

    lcformat_dir : str
        This specifies the directory where the the LC format specification JSON
        produced by this function will be written. By default, this goes to the
        `.astrobase/lcformat-jsons` directory in your home directory.

    Returns
    -------

    str
        Returns the file path to the generated LC format specification JSON
        file.


if isinstance(use_lcformat_dir, str):<EOL><INDENT>lcformat_jsonpath = os.path.join(<EOL>use_lcformat_dir,<EOL>'<STR_LIT>' % formatkey<EOL>)<EOL>if not os.path.exists(lcformat_jsonpath):<EOL><INDENT>lcformat_jsonpath = os.path.join(<EOL>os.path.expanduser('<STR_LIT>'),<EOL>'<STR_LIT>' % formatkey<EOL>)<EOL>if not os.path.exists(lcformat_jsonpath):<EOL><INDENT>install_path = os.path.dirname(__file__)<EOL>install_path = os.path.abspath(<EOL>os.path.join(install_path, '<STR_LIT:..>', '<STR_LIT:data>','<STR_LIT>')<EOL>)<EOL>lcformat_jsonpath = os.path.join(<EOL>install_path,<EOL>'<STR_LIT>' % formatkey<EOL>)<EOL>if not os.path.exists(lcformat_jsonpath):<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% formatkey)<EOL>return None<EOL><DEDENT><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>lcformat_jsonpath = os.path.join(<EOL>os.path.expanduser('<STR_LIT>'),<EOL>'<STR_LIT>' % formatkey<EOL>)<EOL>if not os.path.exists(lcformat_jsonpath):<EOL><INDENT>install_path = os.path.dirname(__file__)<EOL>install_path = os.path.abspath(<EOL>os.path.join(install_path, '<STR_LIT:..>', '<STR_LIT:data>','<STR_LIT>')<EOL>)<EOL>lcformat_jsonpath = os.path.join(<EOL>install_path,<EOL>'<STR_LIT>' % formatkey<EOL>)<EOL>if not os.path.exists(lcformat_jsonpath):<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% formatkey)<EOL>return None<EOL><DEDENT><DEDENT><DEDENT>with open(lcformat_jsonpath) as infd:<EOL><INDENT>lcformatdict = json.load(infd)<EOL><DEDENT>readerfunc_module = lcformatdict['<STR_LIT>']<EOL>readerfunc = lcformatdict['<STR_LIT>']<EOL>readerfunc_kwargs = lcformatdict['<STR_LIT>']<EOL>normfunc_module = lcformatdict['<STR_LIT>']<EOL>normfunc = lcformatdict['<STR_LIT>']<EOL>normfunc_kwargs = lcformatdict['<STR_LIT>']<EOL>fileglob = lcformatdict['<STR_LIT>']<EOL>timecols = lcformatdict['<STR_LIT>']<EOL>magcols = lcformatdict['<STR_LIT>']<EOL>errcols = lcformatdict['<STR_LIT>']<EOL>magsarefluxes = lcformatdict['<STR_LIT>']<EOL>readermodule = _check_extmodule(readerfunc_module, formatkey)<EOL>if not readermodule:<EOL><INDENT>LOGERROR("<STR_LIT>"<EOL>"<STR_LIT>" %<EOL>(readerfunc_module, formatkey))<EOL>return None<EOL><DEDENT>try:<EOL><INDENT>readerfunc_in = getattr(readermodule, readerfunc)<EOL><DEDENT>except AttributeError:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (formatkey, readerfunc_module, readerfunc))<EOL>raise<EOL><DEDENT>if normfunc_module:<EOL><INDENT>normmodule = _check_extmodule(normfunc_module, formatkey)<EOL>if not normmodule:<EOL><INDENT>LOGERROR("<STR_LIT>"<EOL>"<STR_LIT>" %<EOL>(normfunc_module, formatkey))<EOL>return None<EOL><DEDENT><DEDENT>else:<EOL><INDENT>normmodule = None<EOL><DEDENT>if normfunc_module and normfunc:<EOL><INDENT>try:<EOL><INDENT>normfunc_in = getattr(normmodule, normfunc)<EOL><DEDENT>except AttributeError:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (formatkey, normfunc_module, normfunc))<EOL>raise<EOL><DEDENT><DEDENT>else:<EOL><INDENT>normfunc_in = None<EOL><DEDENT>if isinstance(readerfunc_kwargs, dict):<EOL><INDENT>readerfunc_in = partial(readerfunc_in, **readerfunc_kwargs)<EOL><DEDENT>if normfunc_in is not None:<EOL><INDENT>if isinstance(normfunc_kwargs, dict):<EOL><INDENT>normfunc_in = partial(normfunc_in, **normfunc_kwargs)<EOL><DEDENT><DEDENT>returntuple = (<EOL>fileglob,<EOL>readerfunc_in,<EOL>timecols,<EOL>magcols,<EOL>errcols,<EOL>magsarefluxes,<EOL>normfunc_in,<EOL>)<EOL>return returntuple<EOL>

This loads an LC format description from a previously-saved JSON file.

    Parameters
    ----------

    formatkey : str
        The key used to refer to the LC format. This is part of the JSON file's
        name, e.g. the format key 'hat-csv' maps to the format JSON file:
        '<astrobase install path>/data/lcformats/hat-csv.json'.

    use_lcformat_dir : str or None
        If provided, must be the path to a directory that contains the
        corresponding lcformat JSON file for `formatkey`. If this is None, this
        function will look for lcformat JSON files corresponding to the given
        `formatkey`:

        - first, in the directory specified in this kwarg,
        - if not found there, in the home directory: ~/.astrobase/lcformat-jsons
        - if not found there, in: <astrobase install path>/data/lcformats

    Returns
    -------

    tuple
        A tuple of the following form is returned::

            (fileglob       : the file glob of the associated LC files,
             readerfunc_in  : the imported Python function for reading LCs,
             timecols       : list of time col keys to get from the lcdict,
             magcols        : list of mag col keys to get from the lcdict ,
             errcols        : list of err col keys to get from the lcdict,
             magsarefluxes  : True if the measurements are fluxes not mags,
             normfunc_in    : the imported Python function for normalizing LCs)

        All `astrobase.lcproc` functions can then use this tuple to dynamically
        import your LC reader and normalization functions to work with your LC
        format transparently.


try:<EOL><INDENT>formatinfo = get_lcformat(lcformat,<EOL>use_lcformat_dir=lcformatdir)<EOL>if formatinfo:<EOL><INDENT>(dfileglob, readerfunc,<EOL>dtimecols, dmagcols, derrcols,<EOL>magsarefluxes, normfunc) = formatinfo<EOL><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>")<EOL>return None<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION("<STR_LIT>")<EOL>return None<EOL><DEDENT>if timecols is None:<EOL><INDENT>timecols = dtimecols<EOL><DEDENT>if magcols is None:<EOL><INDENT>magcols = dmagcols<EOL><DEDENT>if errcols is None:<EOL><INDENT>errcols = derrcols<EOL><DEDENT>try:<EOL><INDENT>lcdict = readerfunc(lcfile)<EOL>if ( (isinstance(lcdict, (list, tuple))) and<EOL>(isinstance(lcdict[<NUM_LIT:0>], dict)) ):<EOL><INDENT>lcdict = lcdict[<NUM_LIT:0>]<EOL><DEDENT>outfile = os.path.join(outdir, '<STR_LIT>' %<EOL>squeeze(lcdict['<STR_LIT>']).replace('<STR_LIT:U+0020>', '<STR_LIT:->'))<EOL>if excludeprocessed:<EOL><INDENT>test_outfile = os.path.exists(outfile)<EOL>test_outfile_gz = os.path.exists(outfile+'<STR_LIT>')<EOL>if (test_outfile and os.stat(outfile).st_size > <NUM_LIT>):<EOL><INDENT>LOGWARNING('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (lcfile, outfile))<EOL>return outfile<EOL><DEDENT>elif (test_outfile_gz and os.stat(outfile+'<STR_LIT>').st_size > <NUM_LIT>):<EOL><INDENT>LOGWARNING(<EOL>'<STR_LIT>'<EOL>'<STR_LIT>'<EOL>% (lcfile, outfile+'<STR_LIT>')<EOL>)<EOL>return outfile+'<STR_LIT>'<EOL><DEDENT><DEDENT>resultdict = {<EOL>'<STR_LIT>':lcdict['<STR_LIT>'],<EOL>'<STR_LIT>':os.path.basename(lcfile),<EOL>'<STR_LIT>':{'<STR_LIT>':timecols,<EOL>'<STR_LIT>':magcols,<EOL>'<STR_LIT>':errcols,<EOL>'<STR_LIT>':lcformat,<EOL>'<STR_LIT>':lcformatdir,<EOL>'<STR_LIT>':pfmethods,<EOL>'<STR_LIT>':pfkwargs,<EOL>'<STR_LIT>':sigclip,<EOL>'<STR_LIT>':getblssnr}<EOL>}<EOL>if normfunc is not None:<EOL><INDENT>lcdict = normfunc(lcdict)<EOL><DEDENT>for tcol, mcol, ecol in zip(timecols, magcols, errcols):<EOL><INDENT>if '<STR_LIT:.>' in tcol:<EOL><INDENT>tcolget = tcol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>tcolget = [tcol]<EOL><DEDENT>times = _dict_get(lcdict, tcolget)<EOL>if '<STR_LIT:.>' in mcol:<EOL><INDENT>mcolget = mcol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>mcolget = [mcol]<EOL><DEDENT>mags = _dict_get(lcdict, mcolget)<EOL>if '<STR_LIT:.>' in ecol:<EOL><INDENT>ecolget = ecol.split('<STR_LIT:.>')<EOL><DEDENT>else:<EOL><INDENT>ecolget = [ecol]<EOL><DEDENT>errs = _dict_get(lcdict, ecolget)<EOL>if normfunc is None:<EOL><INDENT>ntimes, nmags = normalize_magseries(<EOL>times, mags,<EOL>magsarefluxes=magsarefluxes<EOL>)<EOL>times, mags, errs = ntimes, nmags, errs<EOL><DEDENT>resultdict[mcol] = {}<EOL>finmags = mags[np.isfinite(mags)]<EOL>if finmags.size < minobservations:<EOL><INDENT>LOGERROR('<STR_LIT>'<EOL>'<STR_LIT>'<EOL>'<STR_LIT>' %<EOL>(finmags.size, minobservations, mcol))<EOL>continue<EOL><DEDENT>pfmkeys = []<EOL>for pfmind, pfm, pfkw in zip(range(len(pfmethods)),<EOL>pfmethods,<EOL>pfkwargs):<EOL><INDENT>pf_func = PFMETHODS[pfm]<EOL>pf_kwargs = pfkw<EOL>pf_kwargs.update({'<STR_LIT>':False,<EOL>'<STR_LIT>':nworkers,<EOL>'<STR_LIT>':magsarefluxes,<EOL>'<STR_LIT>':sigclip})<EOL>pfmkey = '<STR_LIT>' % (pfmind, pfm)<EOL>pfmkeys.append(pfmkey)<EOL>resultdict[mcol][pfmkey] = pf_func(<EOL>times, mags, errs,<EOL>**pf_kwargs<EOL>)<EOL><DEDENT>resultdict[mcol]['<STR_LIT>'] = pfmkeys<EOL>if '<STR_LIT>' in pfmethods and getblssnr:<EOL><INDENT>for pfmk in resultdict[mcol]['<STR_LIT>']:<EOL><INDENT>if '<STR_LIT>' in pfmk:<EOL><INDENT>try:<EOL><INDENT>bls = resultdict[mcol][pfmk]<EOL>blssnr = bls_snr(bls, times, mags, errs,<EOL>magsarefluxes=magsarefluxes,<EOL>verbose=False)<EOL>resultdict[mcol][pfmk].update({<EOL>'<STR_LIT>':blssnr['<STR_LIT>'],<EOL>'<STR_LIT>':blssnr['<STR_LIT>'],<EOL>'<STR_LIT>':blssnr['<STR_LIT>'],<EOL>})<EOL>resultdict[mcol][pfmk].update({<EOL>'<STR_LIT>':blssnr['<STR_LIT>'],<EOL>'<STR_LIT>':blssnr['<STR_LIT>']<EOL>})<EOL><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>' %<EOL>lcfile)<EOL>resultdict[mcol][pfmk].update({<EOL>'<STR_LIT>':[np.nan,np.nan,np.nan,np.nan,np.nan],<EOL>'<STR_LIT>':[np.nan,np.nan,np.nan,<EOL>np.nan,np.nan],<EOL>'<STR_LIT>':[np.nan,np.nan,np.nan,<EOL>np.nan,np.nan],<EOL>})<EOL><DEDENT><DEDENT><DEDENT><DEDENT>elif '<STR_LIT>' in pfmethods:<EOL><INDENT>for pfmk in resultdict[mcol]['<STR_LIT>']:<EOL><INDENT>if '<STR_LIT>' in pfmk:<EOL><INDENT>resultdict[mcol][pfmk].update({<EOL>'<STR_LIT>':[np.nan,np.nan,np.nan,np.nan,np.nan],<EOL>'<STR_LIT>':[np.nan,np.nan,np.nan,<EOL>np.nan,np.nan],<EOL>'<STR_LIT>':[np.nan,np.nan,np.nan,<EOL>np.nan,np.nan],<EOL>})<EOL><DEDENT><DEDENT><DEDENT><DEDENT>with open(outfile, '<STR_LIT:wb>') as outfd:<EOL><INDENT>pickle.dump(resultdict, outfd, protocol=pickle.HIGHEST_PROTOCOL)<EOL><DEDENT>return outfile<EOL><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>' % (lcfile, e))<EOL>if raiseonfail:<EOL><INDENT>raise<EOL><DEDENT>return None<EOL><DEDENT>

This runs the period-finding for a single LC.

    Parameters
    ----------

    lcfile : str
        The light curve file to run period-finding on.

    outdir : str
        The output directory where the result pickle will go.

    timecols : list of str or None
        The timecol keys to use from the lcdict in calculating the features.

    magcols : list of str or None
        The magcol keys to use from the lcdict in calculating the features.

    errcols : list of str or None
        The errcol keys to use from the lcdict in calculating the features.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    pfmethods : list of str
        This is a list of period finding methods to run. Each element is a
        string matching the keys of the `PFMETHODS` dict above. By default, this
        runs GLS, PDM, AoVMH, and the spectral window Lomb-Scargle periodogram.

    pfkwargs : list of dicts
        This is used to provide any special kwargs as dicts to each
        period-finding method function specified in `pfmethods`.

    sigclip : float or int or sequence of two floats/ints or None
        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    getblssnr : bool
        If this is True and BLS is one of the methods specified in `pfmethods`,
        will also calculate the stats for each best period in the BLS results:
        transit depth, duration, ingress duration, refit period and epoch, and
        the SNR of the transit.

    nworkers : int
        The number of parallel period-finding workers to launch.

    minobservations : int
        The minimum number of finite LC points required to process a light
        curve.

    excludeprocessed : bool
        If this is True, light curves that have existing period-finding result
        pickles in `outdir` will not be processed.

        FIXME: currently, this uses a dumb method of excluding already-processed
        files. A smarter way to do this is to (i) generate a SHA512 cachekey
        based on a repr of `{'lcfile', 'timecols', 'magcols', 'errcols',
        'lcformat', 'pfmethods', 'sigclip', 'getblssnr', 'pfkwargs'}`, (ii) make
        sure all list kwargs in the dict are sorted, (iii) check if the output
        file has the same cachekey in its filename (last 8 chars of cachekey
        should work), so the result was processed in exactly the same way as
        specifed in the input to this function, and can therefore be
        ignored. Will implement this later.

    raiseonfail : bool
        If something fails and this is True, will raise an Exception instead of
        returning None at the end.

    Returns
    -------

    str
        The path to the output period-finding result pickle.


(lcfile, outdir, timecols, magcols, errcols, lcformat, lcformatdir,<EOL>pfmethods, pfkwargs, getblssnr, sigclip, nworkers, minobservations,<EOL>excludeprocessed) = task<EOL>if os.path.exists(lcfile):<EOL><INDENT>pfresult = runpf(lcfile,<EOL>outdir,<EOL>timecols=timecols,<EOL>magcols=magcols,<EOL>errcols=errcols,<EOL>lcformat=lcformat,<EOL>lcformatdir=lcformatdir,<EOL>pfmethods=pfmethods,<EOL>pfkwargs=pfkwargs,<EOL>getblssnr=getblssnr,<EOL>sigclip=sigclip,<EOL>nworkers=nworkers,<EOL>minobservations=minobservations,<EOL>excludeprocessed=excludeprocessed)<EOL>return pfresult<EOL><DEDENT>else:<EOL><INDENT>LOGERROR('<STR_LIT>' % lcfile)<EOL>return None<EOL><DEDENT>


<EOL>if not os.path.exists(outdir):<EOL><INDENT>os.makedirs(outdir)<EOL><DEDENT>if (liststartindex is not None) and (listmaxobjects is None):<EOL><INDENT>lclist = lclist[liststartindex:]<EOL><DEDENT>elif (liststartindex is None) and (listmaxobjects is not None):<EOL><INDENT>lclist = lclist[:listmaxobjects]<EOL><DEDENT>elif (liststartindex is not None) and (listmaxobjects is not None):<EOL><INDENT>lclist = lclist[liststartindex:liststartindex+listmaxobjects]<EOL><DEDENT>tasklist = [(x, outdir, timecols, magcols, errcols, lcformat, lcformatdir,<EOL>pfmethods, pfkwargs, getblssnr, sigclip, nperiodworkers,<EOL>minobservations,<EOL>excludeprocessed)<EOL>for x in lclist]<EOL>with ProcessPoolExecutor(max_workers=ncontrolworkers) as executor:<EOL><INDENT>resultfutures = executor.map(_runpf_worker, tasklist)<EOL><DEDENT>results = [x for x in resultfutures]<EOL>return results<EOL>

This drives the overall parallel period processing for a list of LCs.

    As a rough benchmark, 25000 HATNet light curves with up to 50000 points per
    LC take about 26 days in total for an invocation of this function using
    GLS+PDM+BLS, 10 periodworkers, and 4 controlworkers (so all 40 'cores') on a
    2 x Xeon E5-2660v3 machine.

    Parameters
    ----------

    lclist : list of str
        The list of light curve file to process.

    outdir : str
        The output directory where the period-finding result pickles will go.

    timecols : list of str or None
        The timecol keys to use from the lcdict in calculating the features.

    magcols : list of str or None
        The magcol keys to use from the lcdict in calculating the features.

    errcols : list of str or None
        The errcol keys to use from the lcdict in calculating the features.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    pfmethods : list of str
        This is a list of period finding methods to run. Each element is a
        string matching the keys of the `PFMETHODS` dict above. By default, this
        runs GLS, PDM, AoVMH, and the spectral window Lomb-Scargle periodogram.

    pfkwargs : list of dicts
        This is used to provide any special kwargs as dicts to each
        period-finding method function specified in `pfmethods`.

    sigclip : float or int or sequence of two floats/ints or None
        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    getblssnr : bool
        If this is True and BLS is one of the methods specified in `pfmethods`,
        will also calculate the stats for each best period in the BLS results:
        transit depth, duration, ingress duration, refit period and epoch, and
        the SNR of the transit.

    nperiodworkers : int
        The number of parallel period-finding workers to launch per object task.

    ncontrolworkers : int
        The number of controlling processes to launch. This effectively sets how
        many objects from `lclist` will be processed in parallel.

    liststartindex : int or None
        This sets the index from where to start in `lclist`.

    listmaxobjects : int or None
        This sets the maximum number of objects in `lclist` to run
        period-finding for in this invocation. Together with `liststartindex`,
        `listmaxobjects` can be used to distribute processing over several
        independent machines if the number of light curves is very large.

    minobservations : int
        The minimum number of finite LC points required to process a light
        curve.

    excludeprocessed : bool
        If this is True, light curves that have existing period-finding result
        pickles in `outdir` will not be processed.

        FIXME: currently, this uses a dumb method of excluding already-processed
        files. A smarter way to do this is to (i) generate a SHA512 cachekey
        based on a repr of `{'lcfile', 'timecols', 'magcols', 'errcols',
        'lcformat', 'pfmethods', 'sigclip', 'getblssnr', 'pfkwargs'}`, (ii) make
        sure all list kwargs in the dict are sorted, (iii) check if the output
        file has the same cachekey in its filename (last 8 chars of cachekey
        should work), so the result was processed in exactly the same way as
        specifed in the input to this function, and can therefore be
        ignored. Will implement this later.

    Returns
    -------

    list of str
        A list of the period-finding pickles created for all of input LCs
        processed.


try:<EOL><INDENT>formatinfo = get_lcformat(lcformat,<EOL>use_lcformat_dir=lcformatdir)<EOL>if formatinfo:<EOL><INDENT>(dfileglob, readerfunc,<EOL>dtimecols, dmagcols, derrcols,<EOL>magsarefluxes, normfunc) = formatinfo<EOL><DEDENT>else:<EOL><INDENT>LOGERROR("<STR_LIT>")<EOL>return None<EOL><DEDENT><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION("<STR_LIT>")<EOL>return None<EOL><DEDENT>if not fileglob:<EOL><INDENT>fileglob = dfileglob<EOL><DEDENT>LOGINFO('<STR_LIT>' % (lcformat, lcdir))<EOL>if recursive is False:<EOL><INDENT>matching = glob.glob(os.path.join(lcdir, fileglob))<EOL><DEDENT>else:<EOL><INDENT>if sys.version_info[:<NUM_LIT:2>] > (<NUM_LIT:3>,<NUM_LIT:4>):<EOL><INDENT>matching = glob.glob(os.path.join(lcdir,<EOL>'<STR_LIT>',<EOL>fileglob),recursive=True)<EOL><DEDENT>else:<EOL><INDENT>walker = os.walk(lcdir)<EOL>matching = []<EOL>for root, dirs, _files in walker:<EOL><INDENT>for sdir in dirs:<EOL><INDENT>searchpath = os.path.join(root,<EOL>sdir,<EOL>fileglob)<EOL>foundfiles = glob.glob(searchpath)<EOL>if foundfiles:<EOL><INDENT>matching.extend(foundfiles)<EOL><DEDENT><DEDENT><DEDENT><DEDENT><DEDENT>if matching and len(matching) > <NUM_LIT:0>:<EOL><INDENT>matching = sorted(matching)<EOL>LOGINFO('<STR_LIT>' % len(matching))<EOL>return parallel_pf(matching,<EOL>outdir,<EOL>timecols=timecols,<EOL>magcols=magcols,<EOL>errcols=errcols,<EOL>lcformat=lcformat,<EOL>lcformatdir=lcformatdir,<EOL>pfmethods=pfmethods,<EOL>pfkwargs=pfkwargs,<EOL>getblssnr=getblssnr,<EOL>sigclip=sigclip,<EOL>nperiodworkers=nperiodworkers,<EOL>ncontrolworkers=ncontrolworkers,<EOL>liststartindex=liststartindex,<EOL>listmaxobjects=listmaxobjects,<EOL>minobservations=minobservations,<EOL>excludeprocessed=excludeprocessed)<EOL><DEDENT>else:<EOL><INDENT>LOGERROR('<STR_LIT>' % (lcformat,<EOL>lcdir))<EOL>return None<EOL><DEDENT>

This runs parallel light curve period finding for directory of LCs.

    Parameters
    ----------

    lcdir : str
        The directory containing the LCs to process.

    outdir : str
        The directory where the resulting period-finding pickles will go.

    fileglob : str or None
        The UNIX file glob to use to search for LCs in `lcdir`. If None, the
        default file glob associated with the registered LC format will be used
        instead.

    recursive : bool
        If True, will search recursively in `lcdir` for light curves to process.

    timecols : list of str or None
        The timecol keys to use from the lcdict in calculating the features.

    magcols : list of str or None
        The magcol keys to use from the lcdict in calculating the features.

    errcols : list of str or None
        The errcol keys to use from the lcdict in calculating the features.

    lcformat : str
        This is the `formatkey` associated with your light curve format, which
        you previously passed in to the `lcproc.register_lcformat`
        function. This will be used to look up how to find and read the light
        curves specified in `basedir` or `use_list_of_filenames`.

    lcformatdir : str or None
        If this is provided, gives the path to a directory when you've stored
        your lcformat description JSONs, other than the usual directories lcproc
        knows to search for them in. Use this along with `lcformat` to specify
        an LC format JSON file that's not currently registered with lcproc.

    pfmethods : list of str
        This is a list of period finding methods to run. Each element is a
        string matching the keys of the `PFMETHODS` dict above. By default, this
        runs GLS, PDM, AoVMH, and the spectral window Lomb-Scargle periodogram.

    pfkwargs : list of dicts
        This is used to provide any special kwargs as dicts to each
        period-finding method function specified in `pfmethods`.

    sigclip : float or int or sequence of two floats/ints or None
        If a single float or int, a symmetric sigma-clip will be performed using
        the number provided as the sigma-multiplier to cut out from the input
        time-series.

        If a list of two ints/floats is provided, the function will perform an
        'asymmetric' sigma-clip. The first element in this list is the sigma
        value to use for fainter flux/mag values; the second element in this
        list is the sigma value to use for brighter flux/mag values. For
        example, `sigclip=[10., 3.]`, will sigclip out greater than 10-sigma
        dimmings and greater than 3-sigma brightenings. Here the meaning of
        "dimming" and "brightening" is set by *physics* (not the magnitude
        system), which is why the `magsarefluxes` kwarg must be correctly set.

        If `sigclip` is None, no sigma-clipping will be performed, and the
        time-series (with non-finite elems removed) will be passed through to
        the output.

    getblssnr : bool
        If this is True and BLS is one of the methods specified in `pfmethods`,
        will also calculate the stats for each best period in the BLS results:
        transit depth, duration, ingress duration, refit period and epoch, and
        the SNR of the transit.

    nperiodworkers : int
        The number of parallel period-finding workers to launch per object task.

    ncontrolworkers : int
        The number of controlling processes to launch. This effectively sets how
        many objects from `lclist` will be processed in parallel.

    liststartindex : int or None
        This sets the index from where to start in `lclist`.

    listmaxobjects : int or None
        This sets the maximum number of objects in `lclist` to run
        period-finding for in this invocation. Together with `liststartindex`,
        `listmaxobjects` can be used to distribute processing over several
        independent machines if the number of light curves is very large.

    minobservations : int
        The minimum number of finite LC points required to process a light
        curve.

    excludeprocessed : bool
        If this is True, light curves that have existing period-finding result
        pickles in `outdir` will not be processed.

        FIXME: currently, this uses a dumb method of excluding already-processed
        files. A smarter way to do this is to (i) generate a SHA512 cachekey
        based on a repr of `{'lcfile', 'timecols', 'magcols', 'errcols',
        'lcformat', 'pfmethods', 'sigclip', 'getblssnr', 'pfkwargs'}`, (ii) make
        sure all list kwargs in the dict are sorted, (iii) check if the output
        file has the same cachekey in its filename (last 8 chars of cachekey
        should work), so the result was processed in exactly the same way as
        specifed in the input to this function, and can therefore be
        ignored. Will implement this later.

    Returns
    -------

    list of str
        A list of the period-finding pickles created for all of input LCs
        processed.


<EOL>with open(xmatchpkl,'<STR_LIT:rb>') as infd:<EOL><INDENT>xmd = pickle.load(infd)<EOL><DEDENT>status_dict = {}<EOL>for cpf in cplist:<EOL><INDENT>cpd = _read_checkplot_picklefile(cpf)<EOL>try:<EOL><INDENT>xmatch_external_catalogs(cpd, xmd,<EOL>xmatchradiusarcsec=xmatchradiusarcsec,<EOL>updatexmatch=updateexisting)<EOL>for xmi in cpd['<STR_LIT>']:<EOL><INDENT>if cpd['<STR_LIT>'][xmi]['<STR_LIT>']:<EOL><INDENT>LOGINFO('<STR_LIT>'<EOL>'<STR_LIT>' %<EOL>(os.path.basename(cpf),<EOL>cpd['<STR_LIT>'],<EOL>cpd['<STR_LIT>'][xmi]['<STR_LIT:name>'],<EOL>cpd['<STR_LIT>'][xmi]['<STR_LIT>']))<EOL><DEDENT>if not resultstodir:<EOL><INDENT>outcpf = _write_checkplot_picklefile(cpd,<EOL>outfile=cpf)<EOL><DEDENT>else:<EOL><INDENT>xcpf = os.path.join(resultstodir, os.path.basename(cpf))<EOL>outcpf = _write_checkplot_picklefile(cpd,<EOL>outfile=xcpf)<EOL><DEDENT><DEDENT>status_dict[cpf] = outcpf<EOL><DEDENT>except Exception as e:<EOL><INDENT>LOGEXCEPTION('<STR_LIT>' % cpf)<EOL>status_dict[cpf] = None<EOL><DEDENT><DEDENT>return status_dict<EOL>

This xmatches external catalogs to a collection of checkplots.

    Parameters
    ----------

    cplist : list of str
        This is the list of checkplot pickle files to process.

    xmatchpkl : str
        The filename of a pickle prepared beforehand with the
        `checkplot.pkl_xmatch.load_xmatch_external_catalogs` function,
        containing collected external catalogs to cross-match the objects in the
        input `cplist` against.

    xmatchradiusarcsec : float
        The match radius to use for the cross-match in arcseconds.

    updateexisting : bool
        If this is True, will only update the `xmatch` dict in each checkplot
        pickle with any new cross-matches to the external catalogs. If False,
        will overwrite the `xmatch` dict with results from the current run.

    resultstodir : str or None
        If this is provided, then it must be a directory to write the resulting
        checkplots to after xmatch is done. This can be used to keep the
        original checkplots in pristine condition for some reason.

    Returns
    -------

    dict
        Returns a dict with keys = input checkplot pickle filenames and vals =
        xmatch status dict for each checkplot pickle.


This xmatches external catalogs to all checkplots in a directory.

    Parameters
    -----------

    cpdir : str
        This is the directory to search in for checkplots.

    xmatchpkl : str
        The filename of a pickle prepared beforehand with the
        `checkplot.pkl_xmatch.load_xmatch_external_catalogs` function,
        containing collected external catalogs to cross-match the objects in the
        input `cplist` against.

    cpfileglob : str
        This is the UNIX fileglob to use in searching for checkplots.

    xmatchradiusarcsec : float
        The match radius to use for the cross-match in arcseconds.

    updateexisting : bool
        If this is True, will only update the `xmatch` dict in each checkplot
        pickle with any new cross-matches to the external catalogs. If False,
        will overwrite the `xmatch` dict with results from the current run.

    resultstodir : str or None
        If this is provided, then it must be a directory to write the resulting
        checkplots to after xmatch is done. This can be used to keep the
        original checkplots in pristine condition for some reason.

    Returns
    -------

    dict
        Returns a dict with keys = input checkplot pickle filenames and vals =
        xmatch status dict for each checkplot pickle.


<EOL>cplist_objectids = []<EOL>cplist_mags = []<EOL>cplist_colors = []<EOL>for cpf in cplist:<EOL><INDENT>cpd = _read_checkplot_picklefile(cpf)<EOL>cplist_objectids.append(cpd['<STR_LIT>'])<EOL>thiscp_mags = []<EOL>thiscp_colors = []<EOL>for cm1, cm2, ym in zip(color_mag1, color_mag2, yaxis_mag):<EOL><INDENT>if (ym in cpd['<STR_LIT>'] and<EOL>cpd['<STR_LIT>'][ym] is not None):<EOL><INDENT>thiscp_mags.append(cpd['<STR_LIT>'][ym])<EOL><DEDENT>else:<EOL><INDENT>thiscp_mags.append(np.nan)<EOL><DEDENT>if (cm1 in cpd['<STR_LIT>'] and<EOL>cpd['<STR_LIT>'][cm1] is not None and<EOL>cm2 in cpd['<STR_LIT>'] and<EOL>cpd['<STR_LIT>'][cm2] is not None):<EOL><INDENT>thiscp_colors.append(cpd['<STR_LIT>'][cm1] -<EOL>cpd['<STR_LIT>'][cm2])<EOL><DEDENT>else:<EOL><INDENT>thiscp_colors.append(np.nan)<EOL><DEDENT><DEDENT>cplist_mags.append(thiscp_mags)<EOL>cplist_colors.append(thiscp_colors)<EOL><DEDENT>cplist_objectids = np.array(cplist_objectids)<EOL>cplist_mags = np.array(cplist_mags)<EOL>cplist_colors = np.array(cplist_colors)<EOL>cmddict = {'<STR_LIT>':cplist_objectids,<EOL>'<STR_LIT>':cplist_mags,<EOL>'<STR_LIT>':cplist_colors,<EOL>'<STR_LIT>':color_mag1,<EOL>'<STR_LIT>':color_mag2,<EOL>'<STR_LIT>':yaxis_mag}<EOL>with open(outpkl,'<STR_LIT:wb>') as outfd:<EOL><INDENT>pickle.dump(cmddict, outfd, pickle.HIGHEST_PROTOCOL)<EOL><DEDENT>plt.close('<STR_LIT:all>')<EOL>return cmddict<EOL>

This makes color-mag diagrams for all checkplot pickles in the provided
    list.

    Can make an arbitrary number of CMDs given lists of x-axis colors and y-axis
    mags to use.

    Parameters
    ----------

    cplist : list of str
        This is the list of checkplot pickles to process.

    outpkl : str
        The filename of the output pickle that will contain the color-mag
        information for all objects in the checkplots specified in `cplist`.

    color_mag1 : list of str
        This a list of the keys in each checkplot's `objectinfo` dict that will
        be used as color_1 in the equation::

                x-axis color = color_mag1 - color_mag2

    color_mag2 : list of str
        This a list of the keys in each checkplot's `objectinfo` dict that will
        be used as color_2 in the equation::

                x-axis color = color_mag1 - color_mag2

    yaxis_mag : list of str
        This is a list of the keys in each checkplot's `objectinfo` dict that
        will be used as the (absolute) magnitude y-axis of the color-mag
        diagrams.

    Returns
    -------

    str
        The path to the generated CMD pickle file for the collection of objects
        in the input checkplot list.

    Notes
    -----

    This can make many CMDs in one go. For example, the default kwargs for
    `color_mag`, `color_mag2`, and `yaxis_mag` result in two CMDs generated and
    written to the output pickle file:

    - CMD1 -> gaiamag - kmag on the x-axis vs gaia_absmag on the y-axis
    - CMD2 -> sdssg - kmag on the x-axis vs rpmj (J reduced PM) on the y-axis