def add_subtract_locally(st, region_depth=<NUM_LIT:3>, filter_size=<NUM_LIT:5>, sigma_cutoff=<NUM_LIT:8>,<EOL>**kwargs):

<EOL>tiles = identify_misfeatured_regions(<EOL>st, filter_size=filter_size, sigma_cutoff=sigma_cutoff)<EOL>n_empty = <NUM_LIT:0><EOL>n_added = <NUM_LIT:0><EOL>new_poses = []<EOL>for t in tiles:<EOL><INDENT>curn, curinds = add_subtract_misfeatured_tile(st, t, **kwargs)<EOL>if curn == <NUM_LIT:0>:<EOL><INDENT>n_empty += <NUM_LIT:1><EOL><DEDENT>else:<EOL><INDENT>n_added += curn<EOL>new_poses.extend(st.obj_get_positions()[curinds])<EOL><DEDENT>if n_empty > region_depth:<EOL><INDENT>break  <EOL><DEDENT><DEDENT>else:  <EOL><INDENT>pass<EOL><DEDENT>return n_added, new_poses<EOL>

Automatically adds and subtracts missing particles based on local
regions of poor fit.

Calls identify_misfeatured_regions to identify regions, then
add_subtract_misfeatured_tile on the tiles in order of size until
region_depth tiles have been checked without adding any particles.

Parameters
----------
st: :class:`peri.states.State`
    The state to add and subtract particles to.
region_depth : Int
    The minimum amount of regions to try; the algorithm terminates if
    region_depth regions have been tried without adding particles.

Other Parameters
----------------
filter_size : Int, optional
    The size of the filter for calculating the local standard deviation;
    should approximately be the size of a poorly featured region in each
    dimension. Best if odd. Default is 5.
sigma_cutoff : Float, optional
    The max allowed deviation of the residuals from what is expected,
    in units of the residuals' standard deviation. Lower means more
    sensitive, higher = less sensitive. Default is 8.0, i.e. one pixel
    out of every ``7*10^11`` is mis-identified randomly. In practice the
    noise is not Gaussian so there are still some regions mis-
    identified as improperly featured.
rad : Float or 'calc', optional
    The initial radius for added particles; added particles radii are
    not fit until the end of add_subtract. Default is ``'calc'``, which
    uses the median radii of active particles.
max_iter : Int, optional
    The maximum number of loops for attempted adds at one tile location.
    Default is 3.
invert : Bool, optional
    Whether to invert the image for feature_guess. Default is ``True``,
    i.e. dark particles on bright background.
max_allowed_remove : Int, optional
    The maximum number of particles to remove. If the misfeatured tile
    contains more than this many particles, raises an error. If it
    contains more than half as many particles, throws a warning. If more
    than this many particles are added, they are optimized in blocks of
    ``max_allowed_remove``. Default is 20.
im_change_frac : Float, between 0 and 1.
    If adding or removing a particle decreases the error less than
    ``im_change_frac *`` the change in the image, the particle is deleted.
    Default is 0.2.
min_derr : Float
    The minimum change in the state's error to keep a particle in the
    image. Default is ``'3sig'`` which uses ``3*st.sigma``.
do_opt : Bool, optional
    Set to False to avoid optimizing particle positions after adding
    them. Default is True
minmass : Float, optional
    The minimum mass for a particle to be identified as a feature, as
    used by trackpy. Defaults to a decent guess.
use_tp : Bool, optional
    Set to True to use trackpy to find missing particles inside the
    image. Not recommended since trackpy deliberately cuts out
    particles at the edge of the image. Default is False.
max_allowed_remove : Int, optional
    The maximum number of particles to remove. If the misfeatured tile
    contains more than this many particles, raises an error. If it
    contains more than half as many particles, throws a warning. If more
    than this many particles are added, they are optimized in blocks of
    ``max_allowed_remove``. Default is 20.

Returns
-------
n_added : Int
    The change in the number of particles; i.e the number added - number
    removed.
new_poses : List
    [N,3] element list of the added particle positions.

Notes
-----
Algorithm Description

1. Identify mis-featured regions by how much the local residuals
   deviate from the global residuals, as measured by the standard
   deviation of both.
2. Loop over each of those regions, and:

   a. Remove every particle in the current region.
   b. Try to add particles in the current region until no more
      can be added while adequately decreasing the error.
   c. Terminate if at least region_depth regions have been
      checked without successfully adding a particle.

Because this algorithm is more judicious about chooosing regions to
check, and more aggressive about removing particles in those regions,
it runs faster and does a better job than the (global) add_subtract.
However, this function usually does not work better as an initial add-
subtract on an image, since (1) it doesn't check for removing small/big
particles per se, and (2) when the poorly-featured regions of the image
are large or when the fit is bad, it will remove essentially all of the
particles, taking a long time. As a result, it's usually best to do a
normal add_subtract first and using this function for tough missing or
double-featured particles.


pos = st.obj_get_positions()<EOL>pxinds_ar = np.round(pos).astype('<STR_LIT:int>')<EOL>inim = st.ishape.translate(-st.pad).contains(pxinds_ar)<EOL>pxinds_tuple = tuple(pxinds_ar[inim].T)<EOL>pxvals = st.data[pxinds_tuple]<EOL>invert = np.median(pxvals) < np.median(st.data)  <EOL>return invert<EOL>

Guesses whether particles are bright on a dark bkg or vice-versa

    Works by checking whether the intensity at the particle centers is
    brighter or darker than the average intensity of the image, by
    comparing the median intensities of each.

    Parameters
    ----------
    st : :class:`peri.states.ImageState`

    Returns
    -------
    invert : bool
        Whether to invert the image for featuring.


def optimize_particle(state, index, method='<STR_LIT>', doradius=True):

blocks = state.param_particle(index)<EOL>if not doradius:<EOL><INDENT>blocks = blocks[:-<NUM_LIT:1>]<EOL><DEDENT>g = state.gradloglikelihood(blocks=blocks)<EOL>if method == '<STR_LIT>':<EOL><INDENT>h = state.jtj(blocks=blocks)<EOL><DEDENT>if method == '<STR_LIT>':<EOL><INDENT>h = state.hessloglikelihood(blocks=blocks)<EOL><DEDENT>step = np.linalg.solve(h, g)<EOL>h = np.zeros_like(g)<EOL>for i in range(len(g)):<EOL><INDENT>state.update(blocks[i], state.state[blocks[i]] - step[i])<EOL><DEDENT>return g,h<EOL>

Methods available are
    gn : Gauss-Newton with JTJ (recommended)
    nr : Newton-Rhaphson with hessian

if doradius, also optimize the radius.


x, y = xy<EOL>r1 = x*x + y<EOL>r2 = y*y + x<EOL>return np.array([r1, r2])<EOL>

Himmelblau's function, as a set of residuals (cost = sum(residuals**2))

The standard Himmelbau's function is with data as [11, 7], and four
minimum at (3.0, 2.0), ~(-2.8, 3.1), ~(-3.8, -3.3), ~(3.6, -1.8).
Himmelblau's function is a quadratic model in both x and y. Its data-
space dimension (2) is equal to its model-space dimension (2), so
there is only parameter-effect curvature.

Parameters
----------
    - xy : 2-element list-like
        The x,y parameters of the model.

Returns
-------
    2-element list-like
        The residuals of the model.
Notes
------
https://en.wikipedia.org/wiki/Himmelblau%27s_function


return xy<EOL>

A simple sphere, as a set of residuals (cost = sum(residuals**2))

Simply returns the input parameters. A linear, uncoupled model.

Parameters
----------
    - xy : 2-element list-like
        The x,y parameters of the model.

Returns
-------
    - xy : 2-element list-like
        The residuals, which are just xy


x, y = xy<EOL>r1 = x<EOL>r2 = A*(y-x*x)<EOL>return np.array([r1, r2])<EOL>

The original rosenbrock banana function, as a set of residuals
(cost = sum(residuals**2))

The original function is with data = [1,0] and A=10, with a global
minimum at (1,1)=(data[0], data[0]^2). It is a coupled model,
quadratic in x and linear in y. Its data-space dimension (2) is equal
to its model-space dimension (2), so there is only parameter-effect
curvature.

Parameters
----------
    - xy : 2-element list-like
        The x,y parameters of the model.

Returns
-------
    2-element list-like
        The residuals of the model.


def rosenbrock_gen(xy, A=<NUM_LIT:10>, order=<NUM_LIT:3>):

x, y = xy<EOL>r1 = x<EOL>r2 = A*(y-x**order / order)<EOL>return np.array([r1, r2])<EOL>

A generalized rosenbrock banana function, as a set of residuals
(cost = sum(residuals**2))

The Rosenbrock function, generalized from a quadratic model to a
higher-order nonlinearity. The residuals are
    r1 = xy[0]
    r2 = xy[1] - xy[0]^n/n
The original function is with data = [1,0] and order=2, and has a
single minimum at (x,y) = (1, 1/order). The model is coupled,
polynomial in x and linear in y. Its data-space dimension (2) is
equal to its model-space dimension (2), so there is only parameter-
effect curvature. (See M Transtrum et al, PRE 2011)

Parameters
----------
    - xy : 2-element list-like
        The x,y parameters of the model.
    - order : Int, optional
        The order of the model nonlinearity. Default is 3

Returns
-------
    2-element list-like
        The residuals of the model.


xp = xd[<NUM_LIT:1>:]<EOL>xi = xd[:-<NUM_LIT:1>]<EOL>r1 = A*(xp - xi*xi)<EOL>r2 = <NUM_LIT:1> - xi<EOL>return np.append(r1, r2)<EOL>

A higher-dimensional modification of the rosenbrock function, as a
set of residuals (cost = sum(residuals**2))

The standard modified function is with data = zeros(2d-2), with a
global minimum at xd = ones(d). It is a coupled model, quadratic in
the first d-1 parameters. Its data-space dimension (2d-2) is greater
than its model-space dimension (d) for d > 2.

Parameters
----------
    - xd : d-element list-like
        The x,y parameters of the model.

Returns
-------
    (2d-2)-element list-like
        The residuals of the model.

Notes
-----
    Based on the multidimensional variation
        f(x) = \sum_{i=1}^{N-1} 100(x_{i+1} - x_i&2) + (1-x_i)^2
This gives a data space of dimension 2*(N-1) =2N-2 and a parameter
space of dimension N. According to wikipedia, has 1 minimia for N=3
at (1,1,1), 2 minima for 4<=N<=7. See:
https://en.wikipedia.org/wiki/Rosenbrock_function#Multidimensional_generalisations


def rosenbrock_gendd(xd, A=<NUM_LIT:10>, order=<NUM_LIT:3>):

xp = xd[<NUM_LIT:1>:]<EOL>xi = xd[:-<NUM_LIT:1>]<EOL>r1 = A*(xp - xi**order)<EOL>r2 = <NUM_LIT:1> - xi<EOL>return np.append(r1, r2)<EOL>

A higher-dimensional modification of a generalized rosenbrock
function, as a set of residuals (cost = sum(residuals**2))

The standard modified function is with data = zeros(N), with a
global minimum at xd=ones(d). This function is a coupled model,
polynomial in the first d-1 parameters. Its data-space dimension
(2d-2) is greater than its model-space dimension (d) for d > 2.

Parameters
----------
    - xd : d-element list-like
        The x,y parameters of the model.
    - order : Int, optional
        The order of the model nonlinearity. Default is 3

Returns
-------
    (2d-2)-element list-like
        The residuals of the model.

Notes
-----
    Based on the multidimensional variation
        f(x) = \sum_{i=1}^{N-1} 100(x_{i+1} - x_i&2) + (1-x_i)^2
This gives a data space of dimension 2*(N-1) =2N-2 and a parameter
space of dimension N. According to wikipedia, the quadratic version
has 1 minimia for N=3 at (1,1,1), 2 minima for 4<=N<=7. See:
https://en.wikipedia.org/wiki/Rosenbrock_function#Multidimensional_generalisations


x, y = xy<EOL>r1 = x - x*y<EOL>r2 = x - x*y*y<EOL>r3 = x - x*y*y*y<EOL>return np.array([r1, r2, r3])<EOL>

The Beale function, as a set of residuals (cost = sum(residuals**2))

The standard Beale's function is with data as [1.5, 2.25, 2.625],
and has a global minima at (3, 0.5). Beale's function is a coupled
model, linear in x and quartic in y. Its data-space dimension (3) is
greater than its model-space dimension (2).

Parameters
----------
    - xy : 2-element list-like
        The x,y parameters of the model.

Returns
-------
    3-element list-like
        The residuals of the model.


x, y = xy<EOL>r1 = x + <NUM_LIT:2>*y<EOL>r2 = y + <NUM_LIT:2>*x<EOL>return np.array([r1, r2])<EOL>

The Booth's function, as a set of residuals (cost = sum(residuals**2))

The standard Booth function is with data as [7, 5], and has a single
global minimum at (1,3). It is a coupled linear model, with the
parameter and data space both 2-dimensional.

Parameters
----------
    - xy : 2-element list-like
        The x,y parameters of the model.

Returns
-------
    2-element list-like
        The residuals of the model.


def tile_overlap(inner, outer, norm=False):

div = <NUM_LIT:1.0>/inner.volume if norm else <NUM_LIT:1.0><EOL>return div*(inner.volume - util.Tile.intersection(inner, outer).volume)<EOL>


def closest_uniform_tile(s, shift, size, other):

region = util.Tile(size, dim=s.dim, dtype='<STR_LIT:float>').translate(shift - s.pad)<EOL>vec = np.round((other.center - region.center) / region.shape)<EOL>return region.translate(region.shape * vec)<EOL>

Given a tiling of space (by state, shift, and size), find the closest
tile to another external tile


def spatial_parameter_groups(s, params, max_mem=<NUM_LIT>, max_decimation=<NUM_LIT:1.0>, **kwargs):

<EOL>

Given a state and a list of parameters, spatially group the parameters into
small regions which have overlapping influence so that they may be optimized
concurrently.

Parameters:
-----------
s : state
    A PERI state which contains parameters

parameters : list of strings
    List of parameters which to group together

Returns:
--------
groups : list of list of strings
    List of grouping of parameters. Each parameter will appear exactly once
    in the resulting groups.


def separate_particles_into_groups(s, region_size=<NUM_LIT>, bounds=None):

imtile = (<EOL>s.oshape.translate(-s.pad) if bounds is None else<EOL>util.Tile(bounds[<NUM_LIT:0>], bounds[<NUM_LIT:1>])<EOL>)<EOL>region = util.Tile(region_size, dim=s.dim)<EOL>trange = np.ceil(imtile.shape.astype('<STR_LIT:float>') / region.shape)<EOL>translations = util.Tile(trange).coords(form='<STR_LIT>')<EOL>translations = translations.reshape(-<NUM_LIT:1>, translations.shape[-<NUM_LIT:1>])<EOL>groups = []<EOL>positions = s.obj_get_positions()<EOL>for v in translations:<EOL><INDENT>tmptile = region.copy().translate(region.shape * v - s.pad)<EOL>groups.append(find_particles_in_tile(positions, tmptile))<EOL><DEDENT>return [g for g in groups if len(g) > <NUM_LIT:0>]<EOL>

Given a state, returns a list of groups of particles. Each group of
particles are located near each other in the image. Every particle
located in the desired region is contained in exactly 1 group.

Parameters:
-----------
s : state
    The PERI state to find particles in.

region_size: int or list of ints
    The size of the box. Groups particles into boxes of shape region_size.
    If region_size is a scalar, the box is a cube of length region_size.
    Default is 40.

bounds: 2-element list-like of 3-element lists.
    The sub-region of the image over which to look for particles.
        bounds[0]: The lower-left  corner of the image region.
        bounds[1]: The upper-right corner of the image region.
    Default (None -> ([0,0,0], s.oshape.shape)) is a box of the entire
    image size, i.e. the default places every particle in the image
    somewhere in the groups.

Returns:
-----------
particle_groups: list
    Each element of particle_groups is an int numpy.ndarray of the
    group of nearby particles. Only contains groups with a nonzero
    number of particles, so the elements don't necessarily correspond
    to a given image region.


default = os.path.join(os.path.expanduser("<STR_LIT>"), "<STR_LIT>")<EOL>return os.environ.get('<STR_LIT>', default)<EOL>

The configuration file either lives in ~/.peri.json or is specified on the
command line via the environment variables PERI_CONF_FILE


return v.lower().replace('<STR_LIT:_>', '<STR_LIT:->').replace('<STR_LIT>', '<STR_LIT>')<EOL>

Translate environment variables to ones corresponding to keys in the
configuration.  In particular, env variables may be made with
"PERI_"+key_name: fftw-threads = PERI_FFTW_THREADS. Each env var
is later checked to see if it has to do with PERI


out = {}<EOL>for k,v in iteritems(os.environ):<EOL><INDENT>if transform(k) in default_conf:<EOL><INDENT>out[transform(k)] = v<EOL><DEDENT><DEDENT>return out<EOL>

Read all environment variables to see if they contain PERI


with open(get_conf_filename(), '<STR_LIT:w>') as f:<EOL><INDENT>json.dump(default_conf, f)<EOL><DEDENT>

Dump the default_conf to the configuration file


try:<EOL><INDENT>conf = copy.copy(default_conf)<EOL>conf.update(json.load(open(get_conf_filename())))<EOL>conf.update(read_environment())<EOL>return conf<EOL><DEDENT>except IOError as e:<EOL><INDENT>create_default_conf()<EOL>return load_conf()<EOL><DEDENT>

Load the configuration with the priority:
    1. environment variables
    2. configuration file
    3. defaults here (default_conf)


def locate_spheres(image, feature_rad, dofilter=False, order=(<NUM_LIT:3> ,<NUM_LIT:3>, <NUM_LIT:3>),<EOL>trim_edge=True, **kwargs):

<EOL>m = models.SmoothFieldModel()<EOL>I = ilms.LegendrePoly2P1D(order=order, constval=image.get_image().mean())<EOL>s = states.ImageState(image, [I], pad=<NUM_LIT:0>, mdl=m)<EOL>if dofilter:<EOL><INDENT>opt.do_levmarq(s, s.params)<EOL><DEDENT>pos = addsub.feature_guess(s, feature_rad, trim_edge=trim_edge, **kwargs)[<NUM_LIT:0>]<EOL>return pos<EOL>

Get an initial featuring of sphere positions in an image.

Parameters
-----------
image : :class:`peri.util.Image` object
    Image object which defines the image file as well as the region.

feature_rad : float
    Radius of objects to find, in pixels. This is a featuring radius
    and not a real radius, so a better value is frequently smaller
    than the real radius (half the actual radius is good). If ``use_tp``
    is True, then the twice ``feature_rad`` is passed as trackpy's
    ``diameter`` keyword.

dofilter : boolean, optional
    Whether to remove the background before featuring. Doing so can
    often greatly increase the success of initial featuring and
    decrease later optimization time. Filtering functions by fitting
    the image to a low-order polynomial and featuring the residuals.
    In doing so, this will change the mean intensity of the featured
    image and hence the good value of ``minmass`` will change when
    ``dofilter`` is True. Default is False.

order : 3-element tuple, optional
    If `dofilter`, the 2+1D Leg Poly approximation to the background
    illumination field. Default is (3,3,3).

Other Parameters
----------------
invert : boolean, optional
    Whether to invert the image for featuring. Set to True if the
    image is dark particles on a bright background. Default is True
minmass : Float or None, optional
    The minimum mass/masscut of a particle. Default is None, which
    calculates internally.
use_tp : Bool, optional
    Whether or not to use trackpy. Default is False, since trackpy
    cuts out particles at the edge.

Returns
--------
positions : np.ndarray [N,3]
    Positions of the particles in order (z,y,x) in image pixel units.

Notes
-----
Optionally filters the image by fitting the image I(x,y,z) to a
polynomial, then subtracts this fitted intensity variation and uses
centroid methods to find the particles.


def get_initial_featuring(statemaker, feature_rad, actual_rad=None,<EOL>im_name=None, tile=None, invert=True, desc='<STR_LIT>', use_full_path=False,<EOL>featuring_params={}, statemaker_kwargs={}, **kwargs):

if actual_rad is None:<EOL><INDENT>actual_rad = feature_rad<EOL><DEDENT>_,  im_name = _pick_state_im_name('<STR_LIT>', im_name, use_full_path=use_full_path)<EOL>im = util.RawImage(im_name, tile=tile)<EOL>pos = locate_spheres(im, feature_rad, invert=invert, **featuring_params)<EOL>if np.size(pos) == <NUM_LIT:0>:<EOL><INDENT>msg = '<STR_LIT>'<EOL>raise ValueError(msg)<EOL><DEDENT>rad = np.ones(pos.shape[<NUM_LIT:0>], dtype='<STR_LIT:float>') * actual_rad<EOL>s = statemaker(im, pos, rad, **statemaker_kwargs)<EOL>RLOG.info('<STR_LIT>')<EOL>if desc is not None:<EOL><INDENT>states.save(s, desc=desc+'<STR_LIT>')<EOL><DEDENT>optimize_from_initial(s, invert=invert, desc=desc, **kwargs)<EOL>return s<EOL>

Completely optimizes a state from an image of roughly monodisperse
particles.

The user can interactively select the image. The state is periodically
saved during optimization, with different filename for different stages
of the optimization.

Parameters
----------
    statemaker : Function
        A statemaker function. Given arguments `im` (a
        :class:`~peri.util.Image`), `pos` (numpy.ndarray), `rad` (ndarray),
        and any additional `statemaker_kwargs`, must return a
        :class:`~peri.states.ImageState`.  There is an example function in
        scripts/statemaker_example.py
    feature_rad : Int, odd
        The particle radius for featuring, as passed to locate_spheres.
    actual_rad : Float, optional
        The actual radius of the particles. Default is feature_rad
    im_name : string, optional
        The file name of the image to load. If not set, it is selected
        interactively through Tk.
    tile : :class:`peri.util.Tile`, optional
        The tile of the raw image to be analyzed. Default is None, the
        entire image.
    invert : Bool, optional
        Whether to invert the image for featuring, as passed to trackpy.
        Default is True.
    desc : String, optional
        A description to be inserted in saved state. The save name will
        be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''
    use_full_path : Bool, optional
        Set to True to use the full path name for the image. Default
        is False.
    featuring_params : Dict, optional
        kwargs-like dict of any additional keyword arguments to pass to
        ``get_initial_featuring``, such as ``'use_tp'`` or ``'minmass'``.
        Default is ``{}``.
    statemaker_kwargs : Dict, optional
        kwargs-like dict of any additional keyword arguments to pass to
        the statemaker function. Default is ``{}``.

Other Parameters
----------------
    max_mem : Numeric
        The maximum additional memory to use for the optimizers, as
        passed to optimize.burn. Default is 1e9.
    min_rad : Float, optional
        The minimum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius smaller than this are identified
        as fake and removed. Default is 0.5 * actual_rad.
    max_rad : Float, optional
        The maximum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius larger than this are identified
        as fake and removed. Default is 1.5 * actual_rad, however you
        may find better results if you make this more stringent.
    rz_order : int, optional
        If nonzero, the order of an additional augmented rscl(z)
        parameter for optimization. Default is 0; i.e. no rscl(z)
        optimization.
    zscale : Float, optional
        The zscale of the image. Default is 1.0

Returns
-------
    s : :class:`peri.states.ImageState`
        The optimized state.

See Also
--------
    feature_from_pos_rad    : Using a previous state's globals and
        user-provided positions and radii as an initial guess,
        completely optimizes a state.

    get_particle_featuring  : Using a previous state's globals and
        positions as an initial guess, completely optimizes a state.

    translate_featuring     : Use a previous state's globals and
        centroids methods for an initial particle guess, completely
        optimizes a state.

Notes
-----
Proceeds by centroid-featuring the image for an initial guess of
particle positions, then optimizing the globals + positions until
termination as called in _optimize_from_centroid.
The ``Other Parameters`` are passed to _optimize_from_centroid.


def feature_from_pos_rad(statemaker, pos, rad, im_name=None, tile=None,<EOL>desc='<STR_LIT>', use_full_path=False, statemaker_kwargs={}, **kwargs):

if np.size(pos) == <NUM_LIT:0>:<EOL><INDENT>raise ValueError('<STR_LIT>')<EOL><DEDENT>elif np.shape(pos)[<NUM_LIT:1>] != <NUM_LIT:3>:<EOL><INDENT>raise ValueError('<STR_LIT>')<EOL><DEDENT>_,  im_name = _pick_state_im_name('<STR_LIT>', im_name, use_full_path=use_full_path)<EOL>im = util.RawImage(im_name, tile=tile)<EOL>s = statemaker(im, pos, rad, **statemaker_kwargs)<EOL>RLOG.info('<STR_LIT>')<EOL>if desc is not None:<EOL><INDENT>states.save(s, desc=desc+'<STR_LIT>')<EOL><DEDENT>optimize_from_initial(s, desc=desc, **kwargs)<EOL>return s<EOL>

Gets a completely-optimized state from an image and an initial guess of
particle positions and radii.

The state is periodically saved during optimization, with different
filename for different stages of the optimization. The user can select
the image.

Parameters
----------
    statemaker : Function
        A statemaker function. Given arguments `im` (a
        :class:`~peri.util.Image`), `pos` (numpy.ndarray), `rad` (ndarray),
        and any additional `statemaker_kwargs`, must return a
        :class:`~peri.states.ImageState`.  There is an example function in
        scripts/statemaker_example.py
    pos : [N,3] element numpy.ndarray.
        The initial guess for the N particle positions.
    rad : N element numpy.ndarray.
        The initial guess for the N particle radii.
    im_name : string or None, optional
        The filename of the image to feature. Default is None, in which
        the user selects the image.
    tile : :class:`peri.util.Tile`, optional
        A tile of the sub-region of the image to feature. Default is
        None, i.e. entire image.
    desc : String, optional
        A description to be inserted in saved state. The save name will
        be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''
    use_full_path : Bool, optional
        Set to True to use the full path name for the image. Default
        is False.
    statemaker_kwargs : Dict, optional
        kwargs-like dict of any additional keyword arguments to pass to
        the statemaker function. Default is ``{}``.

Other Parameters
----------------
    max_mem : Numeric
        The maximum additional memory to use for the optimizers, as
        passed to optimize.burn. Default is 1e9.
    min_rad : Float, optional
        The minimum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius smaller than this are identified
        as fake and removed. Default is 0.5 * actual_rad.
    max_rad : Float, optional
        The maximum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius larger than this are identified
        as fake and removed. Default is 1.5 * actual_rad, however you
        may find better results if you make this more stringent.
    invert : {'guess', True, False}
        Whether to invert the image for featuring, as passed to
        addsubtract.add_subtract. Default is to guess from the
        current state's particle positions.
    rz_order : int, optional
        If nonzero, the order of an additional augmented rscl(z)
        parameter for optimization. Default is 0; i.e. no rscl(z)
        optimization.
    zscale : Float, optional
        The zscale of the image. Default is 1.0

Returns
-------
    s : :class:`peri.states.ImageState`
        The optimized state.

See Also
--------
    get_initial_featuring   : Features an image from scratch, using
        centroid methods as initial particle locations.

    get_particle_featuring  : Using a previous state's globals and
        positions as an initial guess, completely optimizes a state.

    translate_featuring     : Use a previous state's globals and
        centroids methods for an initial particle guess, completely
        optimizes a state.

Notes
-----
The ``Other Parameters`` are passed to _optimize_from_centroid.
Proceeds by centroid-featuring the image for an initial guess of
particle positions, then optimizing the globals + positions until
termination as called in _optimize_from_centroid.


def optimize_from_initial(s, max_mem=<NUM_LIT>, invert='<STR_LIT>', desc='<STR_LIT>', rz_order=<NUM_LIT:3>,<EOL>min_rad=None, max_rad=None):

RLOG.info('<STR_LIT>')<EOL>if desc is not None:<EOL><INDENT>desc_burn = desc + '<STR_LIT>'<EOL>desc_polish = desc + '<STR_LIT>'<EOL><DEDENT>else:<EOL><INDENT>desc_burn, desc_polish = [None] * <NUM_LIT:2><EOL><DEDENT>opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:3>, fractol=<NUM_LIT:0.1>, desc=desc_burn,<EOL>max_mem=max_mem, include_rad=False, dowarn=False)<EOL>opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:3>, fractol=<NUM_LIT:0.1>, desc=desc_burn,<EOL>max_mem=max_mem, include_rad=True, dowarn=False)<EOL>RLOG.info('<STR_LIT>')<EOL>rad = s.obj_get_radii()<EOL>if min_rad is None:<EOL><INDENT>min_rad = <NUM_LIT:0.5> * np.median(rad)<EOL><DEDENT>if max_rad is None:<EOL><INDENT>max_rad = <NUM_LIT> * np.median(rad)<EOL><DEDENT>addsub.add_subtract(s, tries=<NUM_LIT:30>, min_rad=min_rad, max_rad=max_rad,<EOL>invert=invert)<EOL>if desc is not None:<EOL><INDENT>states.save(s, desc=desc + '<STR_LIT>')<EOL><DEDENT>RLOG.info('<STR_LIT>')<EOL>d = opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:8>, fractol=<NUM_LIT>, desc=desc_polish,<EOL>max_mem=max_mem, rz_order=rz_order, dowarn=False)<EOL>if not d['<STR_LIT>']:<EOL><INDENT>RLOG.warn('<STR_LIT>')<EOL><DEDENT>return s<EOL>

Optimizes a state from an initial set of positions and radii, without
any known microscope parameters.

Parameters
----------
    s : :class:`peri.states.ImageState`
        The state to optimize. It is modified internally and returned.
    max_mem : Numeric, optional
        The maximum memory for the optimizer to use. Default is 1e9 (bytes)
    invert : Bool or `'guess'`, optional
        Set to True if the image is dark particles on a bright
        background, False otherwise. Used for add-subtract. The
        default is to guess from the state's current particles.
    desc : String, optional
        An additional description to infix for periodic saving along the
        way. Default is the null string ``''``.
    rz_order : int, optional
        ``rz_order`` as passed to opt.burn. Default is 3
    min_rad : Float or None, optional
        The minimum radius to identify a particles as bad, as passed to
        add-subtract. Default is None, which picks half the median radii.
        If your sample is not monodisperse you should pick a different
        value.
    max_rad : Float or None, optional
        The maximum radius to identify a particles as bad, as passed to
        add-subtract. Default is None, which picks 1.5x the median radii.
        If your sample is not monodisperse you should pick a different
        value.

Returns
-------
    s : :class:`peri.states.ImageState`
        The optimized state, which is the same as the input ``s`` but
        modified in-place.


def translate_featuring(state_name=None, im_name=None, use_full_path=False,<EOL>**kwargs):

state_name, im_name = _pick_state_im_name(<EOL>state_name, im_name, use_full_path=use_full_path)<EOL>s = states.load(state_name)<EOL>im = util.RawImage(im_name, tile=s.image.tile)<EOL>s.set_image(im)<EOL>_translate_particles(s, **kwargs)<EOL>return s<EOL>

Translates one optimized state into another image where the particles
have moved by a small amount (~1 particle radius).

Returns a completely-optimized state. The user can interactively
selects the initial state and the second raw image. The state is
periodically saved during optimization, with different filename for
different stages of the optimization.

Parameters
----------
    state_name : String or None, optional
        The name of the initially-optimized state. Default is None,
        which prompts the user to select the name interactively
        through a Tk window.
    im_name : String or None, optional
        The name of the new image to optimize. Default is None,
        which prompts the user to select the name interactively
        through a Tk window.
    use_full_path : Bool, optional
        Set to True to use the full path of the state instead of
        partial path names (e.g. /full/path/name/state.pkl vs
        state.pkl). Default is False

Other Parameters
----------------
    max_mem : Numeric
        The maximum additional memory to use for the optimizers, as
        passed to optimize.burn. Default is 1e9.
    desc : String, optional
        A description to be inserted in saved state. The save name will
        be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''
    min_rad : Float, optional
        The minimum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius smaller than this are identified
        as fake and removed. Default is 0.5 * actual_rad.
    max_rad : Float, optional
        The maximum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius larger than this are identified
        as fake and removed. Default is 1.5 * actual_rad, however you
        may find better results if you make this more stringent.
    invert : {True, False, 'guess'}
        Whether to invert the image for featuring, as passed to
        addsubtract.add_subtract. Default is to guess from the
        state's current particles.
    rz_order : int, optional
        If nonzero, the order of an additional augmented rscl(z)
        parameter for optimization. Default is 0; i.e. no rscl(z)
        optimization.
    do_polish : Bool, optional
        Set to False to only optimize the particles and add-subtract.
        Default is True, which then runs a polish afterwards.

Returns
-------
    s : :class:`peri.states.ImageState`
        The optimized state.

See Also
--------
    get_initial_featuring   : Features an image from scratch, using
        centroid methods as initial particle locations.

    feature_from_pos_rad    : Using a previous state's globals and
        user-provided positions and radii as an initial guess,
        completely optimizes a state.

    get_particle_featuring  : Using a previous state's globals and
        positions as an initial guess, completely optimizes a state.

Notes
-----
The ``Other Parameters`` are passed to _translate_particles.
Proceeds by:
    1. Optimize particle positions only.
    2. Optimize particle positions and radii only.
    3. Add-subtract missing and bad particles.
    4. If polish, optimize the illumination, background, and particles.
    5. If polish, optimize everything.


def get_particles_featuring(feature_rad, state_name=None, im_name=None,<EOL>use_full_path=False, actual_rad=None, invert=True, featuring_params={},<EOL>**kwargs):

state_name, im_name = _pick_state_im_name(<EOL>state_name, im_name, use_full_path=use_full_path)<EOL>s = states.load(state_name)<EOL>if actual_rad is None:<EOL><INDENT>actual_rad = np.median(s.obj_get_radii())<EOL><DEDENT>im = util.RawImage(im_name, tile=s.image.tile)<EOL>pos = locate_spheres(im, feature_rad, invert=invert, **featuring_params)<EOL>_ = s.obj_remove_particle(np.arange(s.obj_get_radii().size))<EOL>s.obj_add_particle(pos, np.ones(pos.shape[<NUM_LIT:0>])*actual_rad)<EOL>s.set_image(im)<EOL>_translate_particles(s, invert=invert, **kwargs)<EOL>return s<EOL>

Combines centroid featuring with the globals from a previous state.

Runs trackpy.locate on an image, sets the globals from a previous state,
calls _translate_particles

Parameters
----------
    feature_rad : Int, odd
        The particle radius for featuring, as passed to locate_spheres.

    state_name : String or None, optional
        The name of the initially-optimized state. Default is None,
        which prompts the user to select the name interactively
        through a Tk window.
    im_name : String or None, optional
        The name of the new image to optimize. Default is None,
        which prompts the user to select the name interactively
        through a Tk window.
    use_full_path : Bool, optional
        Set to True to use the full path of the state instead of
        partial path names (e.g. /full/path/name/state.pkl vs
        state.pkl). Default is False
    actual_rad : Float or None, optional
        The initial guess for the particle radii. Default is the median
        of the previous state.
    invert : Bool
        Whether to invert the image for featuring, as passed to
        addsubtract.add_subtract and locate_spheres. Set to False
        if the image is bright particles on a dark background.
        Default is True (dark particles on bright background).
    featuring_params : Dict, optional
        kwargs-like dict of any additional keyword arguments to pass to
        ``get_initial_featuring``, such as ``'use_tp'`` or ``'minmass'``.
        Default is ``{}``.

Other Parameters
----------------
    max_mem : Numeric
        The maximum additional memory to use for the optimizers, as
        passed to optimize.burn. Default is 1e9.
    desc : String, optional
        A description to be inserted in saved state. The save name will
        be, e.g., '0.tif-peri-' + desc + 'initial-burn.pkl'. Default is ''
    min_rad : Float, optional
        The minimum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius smaller than this are identified
        as fake and removed. Default is 0.5 * actual_rad.
    max_rad : Float, optional
        The maximum particle radius, as passed to addsubtract.add_subtract.
        Particles with a fitted radius larger than this are identified
        as fake and removed. Default is 1.5 * actual_rad, however you
        may find better results if you make this more stringent.
    rz_order : int, optional
        If nonzero, the order of an additional augmented rscl(z)
        parameter for optimization. Default is 0; i.e. no rscl(z)
        optimization.
    do_polish : Bool, optional
        Set to False to only optimize the particles and add-subtract.
        Default is True, which then runs a polish afterwards.

Returns
-------
    s : :class:`peri.states.ImageState`
        The optimized state.

See Also
--------
    get_initial_featuring   : Features an image from scratch, using
        centroid methods as initial particle locations.

    feature_from_pos_rad    : Using a previous state's globals and
        user-provided positions and radii as an initial guess,
        completely optimizes a state.

    translate_featuring     : Use a previous state's globals and
        centroids methods for an initial particle guess, completely
        optimizes a state.

Notes
-----
    The ``Other Parameters`` are passed to _translate_particles.
Proceeds by:
    1. Find a guess of the particle positions through centroid methods.
    2. Optimize particle positions only.
    3. Optimize particle positions and radii only.
    4. Add-subtract missing and bad particles.
    5. If polish, optimize the illumination, background, and particles.
    6. If polish, optimize everything.


def _pick_state_im_name(state_name, im_name, use_full_path=False):

initial_dir = os.getcwd()<EOL>if (state_name is None) or (im_name is None):<EOL><INDENT>wid = tk.Tk()<EOL>wid.withdraw()<EOL><DEDENT>if state_name is None:<EOL><INDENT>state_name = tkfd.askopenfilename(<EOL>initialdir=initial_dir, title='<STR_LIT>')<EOL>os.chdir(os.path.dirname(state_name))<EOL><DEDENT>if im_name is None:<EOL><INDENT>im_name = tkfd.askopenfilename(<EOL>initialdir=initial_dir, title='<STR_LIT>')<EOL><DEDENT>if (not use_full_path) and (os.path.dirname(im_name) != '<STR_LIT>'):<EOL><INDENT>im_path = os.path.dirname(im_name)<EOL>os.chdir(im_path)<EOL>im_name = os.path.basename(im_name)<EOL><DEDENT>else:<EOL><INDENT>os.chdir(initial_dir)<EOL><DEDENT>return state_name, im_name<EOL>

If state_name or im_name is None, picks them interactively through Tk,
and then sets with or without the full path.

Parameters
----------
    state_name : {string, None}
        The name of the state. If None, selected through Tk.
    im_name : {string, None}
        The name of the image. If None, selected through Tk.
    use_full_path : Bool, optional
        Set to True to return the names as full paths rather than
        relative paths. Default is False (relative path).


def _translate_particles(s, max_mem=<NUM_LIT>, desc='<STR_LIT>', min_rad='<STR_LIT>',<EOL>max_rad='<STR_LIT>', invert='<STR_LIT>', rz_order=<NUM_LIT:0>, do_polish=True):

if desc is not None:<EOL><INDENT>desc_trans = desc + '<STR_LIT>'<EOL>desc_burn = desc + '<STR_LIT>'<EOL>desc_polish = desc + '<STR_LIT>'<EOL><DEDENT>else:<EOL><INDENT>desc_trans, desc_burn, desc_polish = [None]*<NUM_LIT:3><EOL><DEDENT>RLOG.info('<STR_LIT>')<EOL>opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:4>, fractol=<NUM_LIT:0.1>, desc=desc_trans,<EOL>max_mem=max_mem, include_rad=False, dowarn=False)<EOL>opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:4>, fractol=<NUM_LIT>, desc=desc_trans,<EOL>max_mem=max_mem, include_rad=True, dowarn=False)<EOL>RLOG.info('<STR_LIT>')<EOL>addsub.add_subtract(s, tries=<NUM_LIT:30>, min_rad=min_rad, max_rad=max_rad,<EOL>invert=invert)<EOL>if desc is not None:<EOL><INDENT>states.save(s, desc=desc + '<STR_LIT>')<EOL><DEDENT>if do_polish:<EOL><INDENT>RLOG.info('<STR_LIT>')<EOL>opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:3>, fractol=<NUM_LIT>, desc=desc_burn,<EOL>max_mem=max_mem, rz_order=rz_order,dowarn=False)<EOL>RLOG.info('<STR_LIT>')<EOL>d = opt.burn(s, mode='<STR_LIT>', n_loop=<NUM_LIT:4>, fractol=<NUM_LIT>, desc=desc_polish,<EOL>max_mem=max_mem, rz_order=rz_order, dowarn=False)<EOL>if not d['<STR_LIT>']:<EOL><INDENT>RLOG.warn('<STR_LIT>')<EOL><DEDENT><DEDENT>

Workhorse for translating particles. See get_particles_featuring for docs.


<EOL>psf = st.get('<STR_LIT>')<EOL>psf.param_dict['<STR_LIT>'] = psf.param_dict['<STR_LIT>']<EOL>psf.params[psf.params.index('<STR_LIT>')] = '<STR_LIT>'<EOL>psf.global_zscale = True<EOL>psf.param_dict.pop('<STR_LIT>')<EOL>st.trigger_parameter_change()<EOL>st.reset()<EOL>

Links the state ``st`` psf zscale with the global zscale


def finish_state(st, desc='<STR_LIT>', invert='<STR_LIT>'):

for minmass in [None, <NUM_LIT:0>]:<EOL><INDENT>for _ in range(<NUM_LIT:3>):<EOL><INDENT>npart, poses = addsub.add_subtract_locally(st, region_depth=<NUM_LIT:7>,<EOL>minmass=minmass, invert=invert)<EOL>if npart == <NUM_LIT:0>:<EOL><INDENT>break<EOL><DEDENT><DEDENT><DEDENT>opt.finish(st, n_loop=<NUM_LIT:1>, separate_psf=True, desc=desc, dowarn=False)<EOL>opt.burn(st, mode='<STR_LIT>', desc=desc, n_loop=<NUM_LIT:2>, dowarn=False)<EOL>d = opt.finish(st, desc=desc, n_loop=<NUM_LIT:4>, dowarn=False)<EOL>if not d['<STR_LIT>']:<EOL><INDENT>RLOG.warn('<STR_LIT>')<EOL><DEDENT>

Final optimization for the best-possible state.

Runs a local add-subtract to capture any difficult-to-feature particles,
then does another set of optimization designed to get to the best
possible fit.

Parameters
----------
    st : :class:`peri.states.ImageState`
        The state to finish
    desc : String, optional
        Description to intermittently save the state as, as passed to
        state.save. Default is `'finish-state'`.
    invert : {'guess', True, False}
        Whether to invert the image for featuring, as passed to
        addsubtract.add_subtract. Default is to guess from the
        state's current particles.

See Also
--------
    `peri.opt.addsubtract.add_subtract_locally`
    `peri.opt.optimize.finish`


err_message = self.format(record)<EOL>pygmentize(err_message)<EOL>


if wisdomfile is None:<EOL><INDENT>return<EOL><DEDENT>try:<EOL><INDENT>pyfftw.import_wisdom(pickle.load(open(wisdomfile, '<STR_LIT:rb>')))<EOL><DEDENT>except (IOError, TypeError) as e:<EOL><INDENT>log.warn("<STR_LIT>" % wisdomfile)<EOL>save_wisdom(wisdomfile)<EOL><DEDENT>

Prime FFTW with knowledge of which FFTs are best on this machine by
loading 'wisdom' from the file ``wisdomfile``


if wisdomfile is None:<EOL><INDENT>return<EOL><DEDENT>if wisdomfile:<EOL><INDENT>pickle.dump(<EOL>pyfftw.export_wisdom(), open(wisdomfile, '<STR_LIT:wb>'),<EOL>protocol=<NUM_LIT:2><EOL>)<EOL><DEDENT>

Save the acquired 'wisdom' generated by FFTW to file so that future
initializations of FFTW will be faster.


def __init__(self, verbosity='<STR_LIT>', colorlogs=False, logtofile=False,<EOL>logfilename='<STR_LIT>'):

self.log = logging.getLogger('<STR_LIT>')<EOL>self.log.setLevel(<NUM_LIT:1>)<EOL>self.handlers = {}<EOL>self.verbosity = sanitize(verbosity)<EOL>self.logfilename = logfilename<EOL>level = v2l.get(verbosity, '<STR_LIT:info>')<EOL>form  = v2f.get(verbosity, '<STR_LIT>')<EOL>color = '<STR_LIT>' if colorlogs else '<STR_LIT>'<EOL>if logtofile:<EOL><INDENT>self.add_handler(<EOL>name='<STR_LIT>', level=level, formatter=form,<EOL>filename=self.logfilename<EOL>)<EOL><DEDENT>self.add_handler(name=color, level=level, formatter=form)<EOL>

Create a new logger class. Since the logging interface is actually global,
any new logs will create even more clutter on the screen. Therefore,
only create one! (as is created at the bottom of this file)


def set_level(self, level='<STR_LIT:info>', handlers=None):

for h in self.get_handlers(handlers):<EOL><INDENT>h.setLevel(levels[level])<EOL><DEDENT>

Set the logging level (which types of logs are actually printed / recorded)
to one of ['debug', 'info', 'warn', 'error', 'fatal'] in that order
of severity


def set_formatter(self, formatter='<STR_LIT>', handlers=None):

for h in self.get_handlers(handlers):<EOL><INDENT>h.setFormatter(logging.Formatter(formatters[formatter]))<EOL><DEDENT>

Set the text format of messages to one of the pre-determined forms,
one of ['quiet', 'minimal', 'standard', 'verbose']


def add_handler(self, name='<STR_LIT>', level='<STR_LIT:info>', formatter='<STR_LIT>', **kwargs):

<EOL>if name == '<STR_LIT>' and '<STR_LIT:filename>' not in kwargs:<EOL><INDENT>kwargs.update({'<STR_LIT:filename>': self.logfilename})<EOL><DEDENT>if name == '<STR_LIT>' and '<STR_LIT>' not in kwargs:<EOL><INDENT>kwargs.update({'<STR_LIT>': StringIO.StringIO()})<EOL><DEDENT>handler = types[name](**kwargs)<EOL>self.add_handler_raw(handler, name, level=level, formatter=formatter)<EOL>

Add another handler to the logging system if not present already.
Available handlers are currently: ['console-bw', 'console-color', 'rotating-log']


if name in self.handlers:<EOL><INDENT>self.log.removeHandler(self.handlers[name])<EOL><DEDENT>

Remove handler from the logging system if present already.
Available handlers are currently: ['console-bw', 'console-color', 'rotating-log']


try:<EOL><INDENT>formats = {}<EOL>for h in self.get_handlers():<EOL><INDENT>formats[h] = h.formatter<EOL><DEDENT>self.set_formatter(formatter='<STR_LIT>')<EOL>yield<EOL><DEDENT>except Exception as e:<EOL><INDENT>raise<EOL><DEDENT>finally:<EOL><INDENT>for k,v in iteritems(formats):<EOL><INDENT>k.formatter = v<EOL><DEDENT><DEDENT>

Temporarily do not use any formatter so that text printed is raw


def set_verbosity(self, verbosity='<STR_LIT>', handlers=None):

self.verbosity = sanitize(verbosity)<EOL>self.set_level(v2l[verbosity], handlers=handlers)<EOL>self.set_formatter(v2f[verbosity], handlers=handlers)<EOL>

Set the verbosity level of a certain log handler or of all handlers.

Parameters
----------
verbosity : 'v' to 'vvvvv'
    the level of verbosity, more v's is more verbose

handlers : string, or list of strings
    handler names can be found in ``peri.logger.types.keys()``
    Current set is::

        ['console-bw', 'console-color', 'rotating-log']


def pole_removal(noise, poles=None, sig=<NUM_LIT:3>):

center = np.array(noise.shape)/<NUM_LIT:2><EOL>v = np.rollaxis(<EOL>np.array(<EOL>np.meshgrid(*(np.arange(s) for s in noise.shape), indexing='<STR_LIT>')<EOL>), <NUM_LIT:0>, <NUM_LIT:3><EOL>).astype("<STR_LIT:float>")<EOL>filter = np.zeros_like(noise, dtype='<STR_LIT:float>')<EOL>for p in poles:<EOL><INDENT>for pp in [p, center - (p-center)]:<EOL><INDENT>dist = ((v-pp)**<NUM_LIT:2>).sum(axis=-<NUM_LIT:1>)<EOL>filter += np.exp(-dist / (<NUM_LIT:2>*sig**<NUM_LIT:2>))<EOL><DEDENT><DEDENT>filter[filter > <NUM_LIT:1>] = <NUM_LIT:1><EOL>return noise*(<NUM_LIT:1>-filter)<EOL>

Remove the noise poles from a 2d noise distribution to show that affects
the real-space noise picture.

noise -- fftshifted 2d array of q values

poles -- N,2 list of pole locations. the last index is in the order y,x as
determined by mpl interactive plots

for example: poles = np.array([[190,277], [227,253], [233, 256]]


probs = [<NUM_LIT:1>-delete_frac, delete_frac]<EOL>m = np.random.choice([True, False], p.shape[<NUM_LIT:0>], p=probs)<EOL>jumble = np.random.randn(m.sum(), <NUM_LIT:3>)<EOL>return p[m] + jumble<EOL>

randomly deletes particles and adds 1-px noise for a realistic
    initial featuring guess


<EOL>rad = <NUM_LIT:0.5> * np.random.randn(POS.shape[<NUM_LIT:0>]) + <NUM_LIT>  <EOL>part = objs.PlatonicSpheresCollection(POS, rad, zscale=<NUM_LIT>)<EOL>slab = objs.Slab(zpos=<NUM_LIT>, angles=(-<NUM_LIT>, -<NUM_LIT>))<EOL>objects = comp.ComponentCollection([part, slab], category='<STR_LIT>')<EOL>p = exactpsf.FixedSSChebLinePSF(kfki=<NUM_LIT>, zslab=-<NUM_LIT>, alpha=<NUM_LIT>,<EOL>n2n1=<NUM_LIT>, sigkf=-<NUM_LIT>, zscale=<NUM_LIT>, laser_wavelength=<NUM_LIT>)<EOL>i = ilms.BarnesStreakLegPoly2P1D(npts=(<NUM_LIT:16>,<NUM_LIT:10>,<NUM_LIT:8>,<NUM_LIT:4>), zorder=<NUM_LIT:8>)<EOL>b = ilms.LegendrePoly2P1D(order=(<NUM_LIT:7>,<NUM_LIT:2>,<NUM_LIT:2>), category='<STR_LIT>')<EOL>off = comp.GlobalScalar(name='<STR_LIT>', value=-<NUM_LIT>)<EOL>mdl = models.ConfocalImageModel()<EOL>st = states.ImageState(util.NullImage(shape=[<NUM_LIT>,<NUM_LIT:64>,<NUM_LIT:64>]),<EOL>[objects, p, i, b, off], mdl=mdl, model_as_data=True)<EOL>b.update(b.params, BKGVALS)<EOL>i.update(i.params, ILMVALS)<EOL>im = st.model + np.random.randn(*st.model.shape) * <NUM_LIT><EOL>return util.Image(im)<EOL>

Creates an image, as a `peri.util.Image`, which is similar
    to the image in the tutorial


def makestate(im, pos, rad, slab=None, mem_level='<STR_LIT>'):

if slab is not None:<EOL><INDENT>o = comp.ComponentCollection(<EOL>[<EOL>objs.PlatonicSpheresCollection(pos, rad, zscale=zscale),<EOL>slab<EOL>],<EOL>category='<STR_LIT>'<EOL>)<EOL><DEDENT>else:<EOL><INDENT>o = objs.PlatonicSpheresCollection(pos, rad, zscale=zscale)<EOL><DEDENT>p = exactpsf.FixedSSChebLinePSF()<EOL>npts, iorder = _calc_ilm_order(im.get_image().shape)<EOL>i = ilms.BarnesStreakLegPoly2P1D(npts=npts, zorder=iorder)<EOL>b = ilms.LegendrePoly2P1D(order=(<NUM_LIT:9> ,<NUM_LIT:3>, <NUM_LIT:5>), category='<STR_LIT>')<EOL>c = comp.GlobalScalar('<STR_LIT>', <NUM_LIT:0.0>)<EOL>s = states.ImageState(im, [o, i, b, c, p])<EOL>runner.link_zscale(s)<EOL>if mem_level != '<STR_LIT>':<EOL><INDENT>s.set_mem_level(mem_level)<EOL><DEDENT>opt.do_levmarq(s, ['<STR_LIT>'], max_iter=<NUM_LIT:1>, run_length=<NUM_LIT:6>, max_mem=<NUM_LIT>)<EOL>return s<EOL>

Workhorse for creating & optimizing states with an initial centroid
guess.

This is an example function that works for a particular microscope. For
your own microscope, you'll need to change particulars such as the psf
type and the orders of the background and illumination.

Parameters
----------
    im : :class:`~peri.util.RawImage`
        A RawImage of the data.
    pos : [N,3] element numpy.ndarray.
        The initial guess for the N particle positions.
    rad : N element numpy.ndarray.
        The initial guess for the N particle radii.

    slab : :class:`peri.comp.objs.Slab` or None, optional
        If not None, a slab corresponding to that in the image. Default
        is None.
    mem_level : {'lo', 'med-lo', 'med', 'med-hi', 'hi'}, optional
        A valid memory level for the state to control the memory overhead
        at the expense of accuracy. Default is `'hi'`

Returns
-------
    :class:`~peri.states.ImageState`
        An ImageState with a linked z-scale, a ConfocalImageModel, and
        all the necessary components with orders at which are useful for
        my particular test case.


zorder = int(imshape[<NUM_LIT:0>] / <NUM_LIT>) + <NUM_LIT:1><EOL>l_npts = int(imshape[<NUM_LIT:1>] / <NUM_LIT>)+<NUM_LIT:1><EOL>npts = ()<EOL>for a in range(l_npts):<EOL><INDENT>if a < <NUM_LIT:5>:<EOL><INDENT>npts += (int(imshape[<NUM_LIT:2>] * [<NUM_LIT>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT>][a]/<NUM_LIT>) + <NUM_LIT:1>,)<EOL><DEDENT>else:<EOL><INDENT>npts += (int(imshape[<NUM_LIT:2>] * <NUM_LIT:11>/<NUM_LIT>) + <NUM_LIT:1>,)<EOL><DEDENT><DEDENT>return npts, zorder<EOL>

Calculates an ilm order based on the shape of an image. This is based on
something that works for our particular images. Your mileage will vary.

Parameters
----------
    imshape : 3-element list-like
        The shape of the image.

Returns
-------
    npts : tuple
        The number of points to use for the ilm.
    zorder : int
        The order of the z-polynomial.


def diffusion(diffusion_constant=<NUM_LIT>, exposure_time=<NUM_LIT>, samples=<NUM_LIT:200>):

radius = <NUM_LIT:5><EOL>psfsize = np.array([<NUM_LIT>, <NUM_LIT:1.0>, <NUM_LIT>])<EOL>s0 = init.create_single_particle_state(imsize=<NUM_LIT:4>*radius, <EOL>radius=radius, psfargs={'<STR_LIT>': psfsize, '<STR_LIT:error>': <NUM_LIT>})<EOL>finalimage = <NUM_LIT:0>*s0.get_model_image()[s0.inner]<EOL>position = <NUM_LIT:0>*s0.obj.pos[<NUM_LIT:0>]<EOL>for i in range(samples):<EOL><INDENT>offset = np.sqrt(<NUM_LIT:6>*diffusion_constant*exposure_time)*np.random.randn(<NUM_LIT:3>)<EOL>s0.obj.pos[<NUM_LIT:0>] = np.array(s0.image.shape)/<NUM_LIT:2> + offset<EOL>s0.reset()<EOL>finalimage += s0.get_model_image()[s0.inner]<EOL>position += s0.obj.pos[<NUM_LIT:0>]<EOL><DEDENT>finalimage /= float(samples)<EOL>position /= float(samples)<EOL>s = init.create_single_particle_state(imsize=<NUM_LIT:4>*radius, sigma=<NUM_LIT>,<EOL>radius=radius, psfargs={'<STR_LIT>': psfsize, '<STR_LIT:error>': <NUM_LIT>})<EOL>s.reset()<EOL>return s, finalimage, position<EOL>

See `diffusion_correlated` for information related to units, etc


def diffusion_correlated(diffusion_constant=<NUM_LIT>, exposure_time=<NUM_LIT>,<EOL>samples=<NUM_LIT>, phi=<NUM_LIT>):

radius = <NUM_LIT:5><EOL>psfsize = np.array([<NUM_LIT>, <NUM_LIT:1.0>, <NUM_LIT>])/<NUM_LIT:2><EOL>pos, rad, tile = nbody.initialize_particles(N=<NUM_LIT:50>, phi=phi, polydispersity=<NUM_LIT:0.0>)<EOL>sim = nbody.BrownianHardSphereSimulation(<EOL>pos, rad, tile, D=diffusion_constant, dt=exposure_time/samples<EOL>)<EOL>sim.dt = <NUM_LIT><EOL>sim.relax(<NUM_LIT>)<EOL>sim.dt = exposure_time/samples<EOL>c = ((sim.pos - sim.tile.center())**<NUM_LIT:2>).sum(axis=-<NUM_LIT:1>).argmin()<EOL>pc = sim.pos[c].copy()<EOL>sim.pos[c] = sim.pos[<NUM_LIT:0>]<EOL>sim.pos[<NUM_LIT:0>] = pc<EOL>mask = np.zeros_like(sim.rad).astype('<STR_LIT:bool>')<EOL>neigh = sim.neighbors(<NUM_LIT:3>*radius, <NUM_LIT:0>)<EOL>for i in neigh+[<NUM_LIT:0>]:<EOL><INDENT>mask[i] = True<EOL><DEDENT>img = np.zeros(sim.tile.shape)<EOL>s0 = runner.create_state(img, sim.pos, sim.rad, ignoreimage=True)<EOL>finalimage = <NUM_LIT:0>*s0.get_model_image()[s0.inner]<EOL>position = <NUM_LIT:0>*s0.obj.pos<EOL>for i in range(samples):<EOL><INDENT>sim.step(<NUM_LIT:1>, mask=mask)<EOL>s0.obj.pos = sim.pos.copy() + s0.pad<EOL>s0.reset()<EOL>finalimage += s0.get_model_image()[s0.inner]<EOL>position += s0.obj.pos<EOL><DEDENT>finalimage /= float(samples)<EOL>position /= float(samples)<EOL>s = runner.create_state(img, sim.pos, sim.rad, ignoreimage=True)<EOL>s.reset()<EOL>return s, finalimage, position<EOL>

Calculate the (perhaps) correlated diffusion effect between particles
during the exposure time of the confocal microscope. diffusion_constant is
in terms of seconds and pixel sizes exposure_time is in seconds

1 micron radius particle:
    D = kT / (6 a\pi\eta)
    for 80/20 g/w (60 mPas), 3600 nm^2/sec ~ 0.15 px^2/sec
    for 100 % w  (0.9 mPas),               ~ 10.1 px^2/sec
a full 60 layer scan takes 0.1 sec, so a particle is 0.016 sec exposure


def dorun(SNR=<NUM_LIT:20>, ntimes=<NUM_LIT:20>, samples=<NUM_LIT:10>, noise_samples=<NUM_LIT:10>, sweeps=<NUM_LIT:20>, burn=<NUM_LIT:10>,<EOL>correlated=False):

if not correlated:<EOL><INDENT>times = np.logspace(-<NUM_LIT:3>, <NUM_LIT:0>, ntimes)<EOL><DEDENT>else:<EOL><INDENT>times = np.logspace(np.log10(<NUM_LIT>), np.log10(<NUM_LIT:30>), ntimes)<EOL><DEDENT>crbs, vals, errs, poss = [], [], [], []<EOL>for i,t in enumerate(times):<EOL><INDENT>print('<STR_LIT>', i, t)<EOL>for j in range(samples):<EOL><INDENT>print('<STR_LIT:image>', j, '<STR_LIT:|>', end='<STR_LIT:U+0020>') <EOL>if not correlated:<EOL><INDENT>s,im,pos = diffusion(diffusion_constant=<NUM_LIT>, exposure_time=t)<EOL><DEDENT>else:<EOL><INDENT>s,im,pos = diffusion_correlated(diffusion_constant=<NUM_LIT>, exposure_time=t)<EOL><DEDENT>common.set_image(s, im, <NUM_LIT:1.0>/SNR)<EOL>crbs.append(common.crb(s))<EOL>val, err = common.sample(s, im, <NUM_LIT:1.0>/SNR, N=noise_samples, sweeps=sweeps, burn=burn)<EOL>poss.append(pos)<EOL>vals.append(val)<EOL>errs.append(err)<EOL><DEDENT><DEDENT>shape0 = (ntimes, samples, -<NUM_LIT:1>)<EOL>shape1 = (ntimes, samples, noise_samples, -<NUM_LIT:1>)<EOL>crbs = np.array(crbs).reshape(shape0)<EOL>vals = np.array(vals).reshape(shape1)<EOL>errs = np.array(errs).reshape(shape1)<EOL>poss = np.array(poss).reshape(shape0)<EOL>return  [crbs, vals, errs, poss, times]<EOL>

we want to display the errors introduced by pixelation so we plot:
    * CRB, sampled error vs exposure time

a = dorun(ntimes=10, samples=5, noise_samples=5, sweeps=20, burn=8)


def missing_particle(separation=<NUM_LIT:0.0>, radius=RADIUS, SNR=<NUM_LIT:20>):

<EOL>s = init.create_two_particle_state(imsize=<NUM_LIT:6>*radius+<NUM_LIT:4>, axis='<STR_LIT:x>', sigma=<NUM_LIT:1.0>/SNR,<EOL>delta=separation, radius=radius, stateargs={'<STR_LIT>': True}, psfargs={'<STR_LIT:error>': <NUM_LIT>})<EOL>s.obj.typ[<NUM_LIT:1>] = <NUM_LIT:0.><EOL>s.reset()<EOL>return s, s.obj.pos.copy()<EOL>

create a two particle state and compare it to featuring using a single particle guess


def dorun(SNR=<NUM_LIT:20>, sweeps=<NUM_LIT:20>, burn=<NUM_LIT:8>, noise_samples=<NUM_LIT:10>):

radii = np.linspace(<NUM_LIT:2>,<NUM_LIT:10>,<NUM_LIT:8>, endpoint=False)<EOL>crbs, vals, errs = [], [], []<EOL>for radius in radii:<EOL><INDENT>print('<STR_LIT>', radius)<EOL>s,im = pxint(radius=radius, factor=<NUM_LIT:4>)<EOL>goodstate = s.state.copy()<EOL>common.set_image(s, im, <NUM_LIT:1.0>/SNR)<EOL>tcrb = crb(s)<EOL>tval, terr = sample(s, im, <NUM_LIT:1.0>/SNR, N=noise_samples, sweeps=sweeps, burn=burn)<EOL>crbs.append(tcrb)<EOL>vals.append(tval)<EOL>errs.append(terr)<EOL><DEDENT>return np.array(crbs), np.array(vals), np.array(errs), radii<EOL>

we want to display the errors introduced by pixelation so we plot:
    * zero noise, cg image, fit
    * SNR 20, cg image, fit
    * CRB for both

a = dorun(noise_samples=30, sweeps=24, burn=12, SNR=20)


def perfect_platonic_per_pixel(N, R, scale=<NUM_LIT:11>, pos=None, zscale=<NUM_LIT:1.0>, returnpix=None):

<EOL>if scale % <NUM_LIT:2> != <NUM_LIT:1>:<EOL><INDENT>scale += <NUM_LIT:1><EOL><DEDENT>if pos is None:<EOL><INDENT>pos = np.array([(N-<NUM_LIT:1>)/<NUM_LIT>]*<NUM_LIT:3>)<EOL><DEDENT>s = <NUM_LIT:1.0>/scale<EOL>f = zscale**<NUM_LIT:2><EOL>i = pos.astype('<STR_LIT:int>')<EOL>p = i + s*((pos - i)/s).astype('<STR_LIT:int>')<EOL>pos = p + <NUM_LIT> <EOL>image = np.zeros((N,)*<NUM_LIT:3>)<EOL>x,y,z = np.meshgrid(*(range(N),)*<NUM_LIT:3>, indexing='<STR_LIT>')<EOL>for x0,y0,z0 in zip(x.flatten(),y.flatten(),z.flatten()):<EOL><INDENT>ddd = np.sqrt(f*(x0-pos[<NUM_LIT:0>])**<NUM_LIT:2> + (y0-pos[<NUM_LIT:1>])**<NUM_LIT:2> + (z0-pos[<NUM_LIT:2>])**<NUM_LIT:2>)<EOL>if ddd > R + <NUM_LIT:4>:<EOL><INDENT>image[x0,y0,z0] = <NUM_LIT:0.0><EOL>continue<EOL><DEDENT>xp,yp,zp = np.meshgrid(<EOL>*(np.linspace(i-<NUM_LIT:0.5>+s/<NUM_LIT:2>, i+<NUM_LIT:0.5>-s/<NUM_LIT:2>, scale, endpoint=True) for i in (x0,y0,z0)),<EOL>indexing='<STR_LIT>'<EOL>)<EOL>ddd = np.sqrt(f*(xp-pos[<NUM_LIT:0>])**<NUM_LIT:2> + (yp-pos[<NUM_LIT:1>])**<NUM_LIT:2> + (zp-pos[<NUM_LIT:2>])**<NUM_LIT:2>)<EOL>if returnpix is not None and returnpix == [x0,y0,z0]:<EOL><INDENT>outpix = <NUM_LIT:1.0> * (ddd < R)<EOL><DEDENT>vol = (<NUM_LIT:1.0>*(ddd < R) + <NUM_LIT:0.0>*(ddd == R)).sum()<EOL>image[x0,y0,z0] = vol / float(scale**<NUM_LIT:3>)<EOL><DEDENT>if returnpix:<EOL><INDENT>return image, pos, outpix<EOL><DEDENT>return image, pos<EOL>

Create a perfect platonic sphere of a given radius R by supersampling by a
factor scale on a grid of size N.  Scale must be odd.

We are able to perfectly position these particles up to 1/scale. Therefore,
let's only allow those types of shifts for now, but return the actual position
used for the placement.


N = image.shape[<NUM_LIT:0>]<EOL>f = <NUM_LIT:2>*np.pi*np.fft.fftfreq(N)<EOL>kx,ky,kz = np.meshgrid(*(f,)*<NUM_LIT:3>, indexing='<STR_LIT>')<EOL>kv = np.array([kx,ky,kz]).T<EOL>q = np.fft.fftn(image)*np.exp(-<NUM_LIT>*(kv*dx).sum(axis=-<NUM_LIT:1>)).T<EOL>return np.real(np.fft.ifftn(q))<EOL>

Translate an image in fourier-space with plane waves


def doplot(image0, image1, xs, crbs, errors, labels, diff_image_scale=<NUM_LIT:0.1>,<EOL>dolabels=True, multiple_crbs=True, xlim=None, ylim=None, highlight=None,<EOL>detailed_labels=False, xlabel="<STR_LIT>", title="<STR_LIT>"):

if len(crbs) != len(errors) or len(crbs) != len(labels):<EOL><INDENT>raise IndexError("<STR_LIT>")<EOL><DEDENT>fig = pl.figure(figsize=(<NUM_LIT>,<NUM_LIT:7>))<EOL>ax = fig.add_axes([<NUM_LIT>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT>])<EOL>gs = ImageGrid(fig, rect=[<NUM_LIT>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT>], nrows_ncols=(<NUM_LIT:2>,<NUM_LIT:1>), axes_pad=<NUM_LIT>,<EOL>cbar_location='<STR_LIT:right>', cbar_mode='<STR_LIT>', cbar_size='<STR_LIT>', cbar_pad=<NUM_LIT>)<EOL>diffm = diff_image_scale*np.ceil(np.abs(image1).max()/diff_image_scale)<EOL>im0 = gs[<NUM_LIT:0>].imshow(image0, vmin=<NUM_LIT:0>, vmax=<NUM_LIT:1>, cmap='<STR_LIT>')<EOL>im1 = gs[<NUM_LIT:1>].imshow(image1, vmin=-diffm, vmax=diffm, cmap='<STR_LIT>')<EOL>cb0 = pl.colorbar(im0, cax=gs[<NUM_LIT:0>].cax, ticks=[<NUM_LIT:0>,<NUM_LIT:1>])<EOL>cb1 = pl.colorbar(im1, cax=gs[<NUM_LIT:1>].cax, ticks=[-diffm,diffm]) <EOL>cb0.ax.set_yticklabels(['<STR_LIT:0>', '<STR_LIT:1>'])<EOL>cb1.ax.set_yticklabels(['<STR_LIT>' % diffm, '<STR_LIT>' % diffm])<EOL>image_names = ["<STR_LIT>", "<STR_LIT>"]<EOL>for i in range(<NUM_LIT:2>):<EOL><INDENT>gs[i].set_xticks([])<EOL>gs[i].set_yticks([])<EOL>gs[i].set_ylabel(image_names[i])<EOL>if dolabels:<EOL><INDENT>lbl(gs[i], figlbl[i])<EOL><DEDENT><DEDENT>symbols = ['<STR_LIT:o>', '<STR_LIT>', '<STR_LIT:D>', '<STR_LIT:>>']<EOL>for i in range(len(labels)):<EOL><INDENT>c = COLORS[i]<EOL>if multiple_crbs or i == <NUM_LIT:0>:<EOL><INDENT>if multiple_crbs:<EOL><INDENT>label = labels[i] if (i != <NUM_LIT:0> and not detailed_labels) else '<STR_LIT>' % labels[i]<EOL><DEDENT>else:<EOL><INDENT>label = '<STR_LIT>'<EOL><DEDENT>ax.plot(xs[i], crbs[i], '<STR_LIT:->', c=c, lw=<NUM_LIT:3>, label=label)<EOL><DEDENT>label = labels[i] if (i != <NUM_LIT:0> and not detailed_labels) else '<STR_LIT>' % labels[i]<EOL>ax.plot(xs[i], errors[i], symbols[i], ls='<STR_LIT>', lw=<NUM_LIT:2>, c=c, label=label, ms=<NUM_LIT:12>)<EOL><DEDENT>if dolabels:<EOL><INDENT>lbl(ax, '<STR_LIT:D>')<EOL><DEDENT>ax.loglog()<EOL>if xlim:<EOL><INDENT>ax.set_xlim(xlim)<EOL><DEDENT>if ylim:<EOL><INDENT>ax.set_ylim(ylim)<EOL><DEDENT>ax.legend(loc='<STR_LIT>', ncol=<NUM_LIT:2>, prop={'<STR_LIT:size>': <NUM_LIT>}, numpoints=<NUM_LIT:1>)<EOL>ax.set_xlabel(xlabel)<EOL>ax.set_ylabel(r"<STR_LIT>")<EOL>ax.grid(False, which='<STR_LIT>', axis='<STR_LIT>')<EOL>ax.set_title(title)<EOL>return gs, ax<EOL>

Standardizing the plot format of the does_matter section.  See any of the
accompaning files to see how to use this generalized plot.

image0 : ground true
image1 : difference image
xs : list of x values for the plots
crbs : list of lines of values of the crbs
errors : list of lines of errors
labels : legend labels for each curve


def create_comparison_state(image, position, radius=<NUM_LIT>, snr=<NUM_LIT:20>,<EOL>method='<STR_LIT>', extrapad=<NUM_LIT:2>, zscale=<NUM_LIT:1.0>):

<EOL>image = common.pad(image, extrapad, <NUM_LIT:0>)<EOL>s = init.create_single_particle_state(imsize=np.array(image.shape), sigma=<NUM_LIT:1.0>/snr,<EOL>radius=radius, psfargs={'<STR_LIT>': np.array([<NUM_LIT>, <NUM_LIT:1.0>, <NUM_LIT>]), '<STR_LIT:error>': <NUM_LIT>, '<STR_LIT>': <NUM_LIT:2>},<EOL>objargs={'<STR_LIT>': method}, stateargs={'<STR_LIT>': False, '<STR_LIT>': <NUM_LIT:4>, '<STR_LIT>': zscale})<EOL>s.obj.pos[<NUM_LIT:0>] = position + s.pad + extrapad<EOL>s.reset()<EOL>s.model_to_true_image()<EOL>timage = <NUM_LIT:1>-np.pad(image, s.pad, mode='<STR_LIT>', constant_values=<NUM_LIT:0>)<EOL>timage = s.psf.execute(timage)<EOL>return s, timage[s.inner]<EOL>

Take a platonic image and position and create a state which we can
use to sample the error for peri. Also return the blurred platonic
image so we can vary the noise on it later


def dorun(method, platonics=None, nsnrs=<NUM_LIT:20>, noise_samples=<NUM_LIT:30>, sweeps=<NUM_LIT:30>, burn=<NUM_LIT:15>):

sigmas = np.logspace(np.log10(<NUM_LIT:1.0>/<NUM_LIT>), <NUM_LIT:0>, nsnrs)<EOL>crbs, vals, errs, poss = [], [], [], []<EOL>for sigma in sigmas:<EOL><INDENT>print("<STR_LIT>", sigma)<EOL>for i, (image, pos) in enumerate(platonics):<EOL><INDENT>print('<STR_LIT:image>', i, '<STR_LIT:|>', end='<STR_LIT:U+0020>') <EOL>s,im = create_comparison_state(image, pos, method=method)<EOL>set_image(s, im, sigma)<EOL>crbs.append(crb(s))<EOL>val, err = sample(s, im, sigma, N=noise_samples, sweeps=sweeps, burn=burn)<EOL>poss.append(pos)<EOL>vals.append(val)<EOL>errs.append(err)<EOL><DEDENT><DEDENT>shape0 = (nsnrs, len(platonics), -<NUM_LIT:1>)<EOL>shape1 = (nsnrs, len(platonics), noise_samples, -<NUM_LIT:1>)<EOL>crbs = np.array(crbs).reshape(shape0)<EOL>vals = np.array(vals).reshape(shape1)<EOL>errs = np.array(errs).reshape(shape1)<EOL>poss = np.array(poss).reshape(shape0)<EOL>return  [crbs, vals, errs, poss, sigmas]<EOL>

platonics = create_many_platonics(N=50)
dorun(platonics)


psfsize = np.array([<NUM_LIT>, <NUM_LIT:1.0>, <NUM_LIT>])<EOL>s0 = init.create_single_particle_state(imsize=<NUM_LIT:4>*radius, <EOL>radius=radius, psfargs={'<STR_LIT>': psfsize, '<STR_LIT:error>': <NUM_LIT>})<EOL>sl = np.s_[s0.pad:-s0.pad,s0.pad:-s0.pad,s0.pad:-s0.pad]<EOL>finalimage = <NUM_LIT:0>*s0.get_model_image()[sl]<EOL>position = <NUM_LIT:0>*s0.obj.pos[<NUM_LIT:0>]<EOL>for i in range(finalimage.shape[<NUM_LIT:0>]):<EOL><INDENT>offset = jitter*np.random.randn(<NUM_LIT:3>)*np.array([<NUM_LIT:1>,<NUM_LIT:0>,<NUM_LIT:0>])<EOL>s0.obj.pos[<NUM_LIT:0>] = np.array(s0.image.shape)/<NUM_LIT:2> + offset<EOL>s0.reset()<EOL>finalimage[i] = s0.get_model_image()[sl][i]<EOL>position += s0.obj.pos[<NUM_LIT:0>]<EOL><DEDENT>position /= float(finalimage.shape[<NUM_LIT:0>])<EOL>s = init.create_single_particle_state(imsize=<NUM_LIT:4>*radius, sigma=<NUM_LIT>,<EOL>radius=radius, psfargs={'<STR_LIT>': psfsize, '<STR_LIT:error>': <NUM_LIT>})<EOL>s.reset()<EOL>return s, finalimage, position<EOL>

scan jitter is in terms of the fractional pixel difference when
moving the laser in the z-direction


def dorun(SNR=<NUM_LIT:20>, njitters=<NUM_LIT:20>, samples=<NUM_LIT:10>, noise_samples=<NUM_LIT:10>, sweeps=<NUM_LIT:20>, burn=<NUM_LIT:10>):

jitters = np.logspace(-<NUM_LIT:6>, np.log10(<NUM_LIT:0.5>), njitters)<EOL>crbs, vals, errs, poss = [], [], [], []<EOL>for i,t in enumerate(jitters):<EOL><INDENT>print('<STR_LIT>', i, t)<EOL>for j in range(samples):<EOL><INDENT>print('<STR_LIT:image>', j, '<STR_LIT:|>', end='<STR_LIT:U+0020>') <EOL>s,im,pos = zjitter(jitter=t)<EOL>common.set_image(s, im, <NUM_LIT:1.0>/SNR)<EOL>crbs.append(crb(s))<EOL>val, err = sample(s, im, <NUM_LIT:1.0>/SNR, N=noise_samples, sweeps=sweeps, burn=burn)<EOL>poss.append(pos)<EOL>vals.append(val)<EOL>errs.append(err)<EOL><DEDENT><DEDENT>shape0 = (njitters, samples, -<NUM_LIT:1>)<EOL>shape1 = (njitters, samples, noise_samples, -<NUM_LIT:1>)<EOL>crbs = np.array(crbs).reshape(shape0)<EOL>vals = np.array(vals).reshape(shape1)<EOL>errs = np.array(errs).reshape(shape1)<EOL>poss = np.array(poss).reshape(shape0)<EOL>return  [crbs, vals, errs, poss, jitters]<EOL>


def fit_edge(separation, radius=<NUM_LIT>, samples=<NUM_LIT:100>, imsize=<NUM_LIT:64>, sigma=<NUM_LIT>, axis='<STR_LIT:z>'):

terrors = []<EOL>berrors = []<EOL>crbs = []<EOL>for sep in separation:<EOL><INDENT>print('<STR_LIT:=>'*<NUM_LIT>)<EOL>print('<STR_LIT>', sep, end='<STR_LIT:U+0020>')<EOL>s = init.create_two_particle_state(imsize, radius=radius, delta=sep, sigma=<NUM_LIT>,<EOL>axis='<STR_LIT:z>', psfargs={'<STR_LIT>': (<NUM_LIT>,<NUM_LIT:1.0>,<NUM_LIT>), '<STR_LIT:error>': <NUM_LIT>},<EOL>stateargs={'<STR_LIT>': True, '<STR_LIT>': const.PAD})<EOL>d = np.array([<NUM_LIT:0>,<NUM_LIT:0.5>,<NUM_LIT:0.5>])<EOL>s.obj.pos -= d<EOL>s.reset()<EOL>bl = s.blocks_particle(<NUM_LIT:0>)<EOL>s.update(bl[<NUM_LIT:0>], np.array([s.pad+radius]))<EOL>bl = s.blocks_particle(<NUM_LIT:1>)<EOL>s.update(bl[<NUM_LIT:0>], np.array([s.pad-radius]))<EOL>if axis == '<STR_LIT:z>':<EOL><INDENT>bl = s.blocks_particle(<NUM_LIT:1>)<EOL>s.update(bl[<NUM_LIT:0>], s.state[bl[<NUM_LIT:0>]]-sep)<EOL>s.model_to_true_image()<EOL><DEDENT>if axis == '<STR_LIT>':<EOL><INDENT>bl = s.blocks_particle(<NUM_LIT:1>)<EOL>s.update(bl[<NUM_LIT:2>], s.state[bl[<NUM_LIT:2>]]+sep)<EOL>s.model_to_true_image()<EOL><DEDENT>p = s.state[s.b_pos].reshape(-<NUM_LIT:1>,<NUM_LIT:3>).copy()<EOL>print(p[<NUM_LIT:0>], p[<NUM_LIT:1>])<EOL>bl = s.explode(s.b_pos)<EOL>crbs.append(np.sqrt(np.diag(np.linalg.inv(s.fisher_information(blocks=bl)))).reshape(-<NUM_LIT:1>,<NUM_LIT:3>))<EOL>tmp_tp, tmp_bf = [],[]<EOL>for i in range(samples):<EOL><INDENT>print(i)<EOL>bench.jiggle_particles(s, pos=p, sig=<NUM_LIT>, mask=np.array([<NUM_LIT:1>,<NUM_LIT:1>,<NUM_LIT:1>]))<EOL>t = bench.trackpy(s)<EOL>b = bench.bamfpy_positions(s, sweeps=<NUM_LIT:15>)<EOL>tmp_tp.append(bench.error(s, t))<EOL>tmp_bf.append(bench.error(s, b))<EOL><DEDENT>terrors.append(tmp_tp)<EOL>berrors.append(tmp_bf)<EOL><DEDENT>return np.array(crbs), np.array(terrors), np.array(berrors)<EOL>

axis is 'z' or 'xy'
seps = np.linspace(0,2,20) 'z'
seps = np.linspace(-2,2,20) 'xy'


return self<EOL>

Since we wish to use the GaussianPSF in the model by calling P(D), the
get function will simply return this object and we will override
__call__ so that we can use P(...).


<EOL>tile = peri.util.Tile(field.shape)<EOL>rx, ry = tile.kvectors(norm=<NUM_LIT:1.0>/tile.shape)<EOL>sx, sy = self.values<EOL>psf = np.exp(-((rx/sx)**<NUM_LIT:2> + (ry/sy)**<NUM_LIT:2>)/<NUM_LIT:2>)<EOL>psf = psf / psf.sum()<EOL>self.psf = psf<EOL>out = fft.fftn(fft.ifftn(field)*fft.ifftn(psf))<EOL>return fftnorm(np.real(out))<EOL>

Accept a field, apply the point-spread-function, and return
the resulting image of a blurred field


Abstract interface for chunker creation.


return DefaultMultiLevelChunker(chunk_sizes, self.create_chunker)<EOL>

Create a multi-level chunker performing chunking with different chunk sizes.

        Args:
            chunk_sizes (list): List of target chunk sizes.

                Warning:
                    For performance reasons, behavior is only defined if chunk sizes are passed in order, i.e., from
                    lowest to highest value.

        Returns:
            BaseMultiLevelChunker: A multi-level chunker object.


@abc.abstractmethod<EOL><INDENT>def next_chunk_boundaries(self, buf, prepend_bytes=<NUM_LIT:0>):<DEDENT>

Computes the next chunk boundaries within `buf`.

        Note:
            If called more than once, output depends on `all` previous calls of this function: The chunking algorithm is
            applied to the concatenation of all `buf` values.

        Args:
            buf (bytes): The message that is to be chunked.
            prepend_bytes (Optional[int]): Optional number of zero bytes that should be input to the chunking algorithm
                before `buf`.

        Returns:
            iterable: An iterable yielding chunk boundary positions relative to `buf`.


def next_chunk_boundaries(self, buf, prepend_bytes=<NUM_LIT:0>):

return (boundary for boundary, _ in self.next_chunk_boundaries_levels(buf, prepend_bytes))<EOL>

Computes the next chunk boundaries within `buf`.

        See :meth:`.BaseChunker.next_chunk_boundaries`.


@abc.abstractmethod<EOL><INDENT>def next_chunk_boundaries_levels(self, buf, prepend_bytes=<NUM_LIT:0>):<DEDENT>

Computes the next chunk boundaries within `buf`.

        Similar to :meth:`.next_chunk_boundaries`, but information about which chunker led to a respective boundary is
        included in the returned value.

        Args:
            buf (bytes): The message that is to be chunked.
            prepend_bytes (Optional[int]): Optional number of zero bytes that should be input to the chunking algorithm
                before `buf`.

        Returns:
            iterable: An iterable yielding tuples (boundary, level), where boundary is a boundary position relative to
                `buf` and level is the index of the chunker (i.e., the index of its chunk size specified during
                instantiation) that yielded the boundary.

            If multiple chunkers yield the same boundary, it is returned only once, along with the highest matching
            chunker index.


def next_chunk_boundaries_levels(self, buf, prepend_bytes=<NUM_LIT:0>):

boundaries = {}<EOL>for level_index, chunker in enumerate(self._chunkers):<EOL><INDENT>boundaries.update(<EOL>dict([(boundary, level_index) for boundary in chunker.next_chunk_boundaries(buf, prepend_bytes)]))<EOL><DEDENT>return sorted(boundaries.items())<EOL>

Computes the next chunk boundaries within `buf`.

        Similar to :meth:`.next_chunk_boundaries`, but information about which chunker led to a respective boundary is
        included in the returned value.

        Args:
            buf (bytes): The message that is to be chunked.
            prepend_bytes (Optional[int]): Optional number of zero bytes that should be input to the chunking algorithm
                before `buf`.

        Returns:
            list: List of tuples (boundary, level), where boundary is a boundary position relative to `buf` and level is
                the index of the chunker (i.e., the index of its chunk size specified during instantiation) that yielded
                the boundary.

            If multiple chunkers yield the same boundary, it is returned only once, along with the highest matching
            chunker index.


return SC._Chunker(chunk_size)<EOL>

Create a chunker performing static chunking (SC) with a specific chunk size.

        Args:
            chunk_size (int): Target chunk size.

        Returns:
            BaseChunker: A chunker object.


rolling_hash = _rabinkarprh.RabinKarpHash(self.window_size, self._seed)<EOL>rolling_hash.set_threshold(<NUM_LIT:1.0> / chunk_size)<EOL>return RabinKarpCDC._Chunker(rolling_hash)<EOL>

Create a chunker performing content-defined chunking (CDC) using Rabin Karp's rolling hash scheme with a
        specific, expected chunk size.

        Args:
            chunk_size (int): (Expected) target chunk size.

        Returns:
            BaseChunker: A chunker object.


rolling_hash = _rabinkarprh.RabinKarpMultiThresholdHash(self.window_size, self._seed,<EOL>[<NUM_LIT:1.0> / chunk_size for chunk_size in chunk_sizes])<EOL>return RabinKarpCDC._MultiLevelChunker(rolling_hash)<EOL>

Create a multi-level chunker performing content-defined chunking (CDC) using Rabin Karp's rolling hash scheme
        with different specific, expected chunk sizes.

        Args:
            chunk_sizes (list): List of (expected) target chunk sizes.

                Warning:
                    For performance reasons, behavior is only defined if chunk sizes are passed in order, i.e., from
                    lowest to highest value.

        Returns:
            BaseMultiLevelChunker: A multi-level chunker object.


def cleanup_old_versions(<EOL>src, keep_last_versions,<EOL>config_file='<STR_LIT>', profile_name=None,<EOL>):

if keep_last_versions <= <NUM_LIT:0>:<EOL><INDENT>print("<STR_LIT>")<EOL><DEDENT>else:<EOL><INDENT>path_to_config_file = os.path.join(src, config_file)<EOL>cfg = read_cfg(path_to_config_file, profile_name)<EOL>profile_name = cfg.get('<STR_LIT>')<EOL>aws_access_key_id = cfg.get('<STR_LIT>')<EOL>aws_secret_access_key = cfg.get('<STR_LIT>')<EOL>client = get_client(<EOL>'<STR_LIT>', profile_name, aws_access_key_id, aws_secret_access_key,<EOL>cfg.get('<STR_LIT>'),<EOL>)<EOL>response = client.list_versions_by_function(<EOL>FunctionName=cfg.get('<STR_LIT>'),<EOL>)<EOL>versions = response.get('<STR_LIT>')<EOL>if len(response.get('<STR_LIT>')) < keep_last_versions:<EOL><INDENT>print('<STR_LIT>')<EOL><DEDENT>else:<EOL><INDENT>version_numbers = [elem.get('<STR_LIT>') for elem in<EOL>versions[<NUM_LIT:1>:-keep_last_versions]]<EOL>for version_number in version_numbers:<EOL><INDENT>try:<EOL><INDENT>client.delete_function(<EOL>FunctionName=cfg.get('<STR_LIT>'),<EOL>Qualifier=version_number,<EOL>)<EOL><DEDENT>except botocore.exceptions.ClientError as e:<EOL><INDENT>print('<STR_LIT>'<EOL>.format(version_number, e.message))<EOL><DEDENT><DEDENT><DEDENT><DEDENT>

Deletes old deployed versions of the function in AWS Lambda.

    Won't delete $Latest and any aliased version

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param int keep_last_versions:
        The number of recent versions to keep and not delete


def deploy(<EOL>src, requirements=None, local_package=None,<EOL>config_file='<STR_LIT>', profile_name=None,<EOL>preserve_vpc=False<EOL>):

<EOL>path_to_config_file = os.path.join(src, config_file)<EOL>cfg = read_cfg(path_to_config_file, profile_name)<EOL>path_to_zip_file = build(<EOL>src, config_file=config_file,<EOL>requirements=requirements,<EOL>local_package=local_package,<EOL>)<EOL>existing_config = get_function_config(cfg)<EOL>if existing_config:<EOL><INDENT>update_function(cfg, path_to_zip_file, existing_config, preserve_vpc=preserve_vpc)<EOL><DEDENT>else:<EOL><INDENT>create_function(cfg, path_to_zip_file)<EOL><DEDENT>

Deploys a new function to AWS Lambda.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)


def deploy_s3(<EOL>src, requirements=None, local_package=None,<EOL>config_file='<STR_LIT>', profile_name=None,<EOL>preserve_vpc=False<EOL>):

<EOL>path_to_config_file = os.path.join(src, config_file)<EOL>cfg = read_cfg(path_to_config_file, profile_name)<EOL>path_to_zip_file = build(<EOL>src, config_file=config_file, requirements=requirements,<EOL>local_package=local_package,<EOL>)<EOL>use_s3 = True<EOL>s3_file = upload_s3(cfg, path_to_zip_file, use_s3)<EOL>existing_config = get_function_config(cfg)<EOL>if existing_config:<EOL><INDENT>update_function(cfg, path_to_zip_file, existing_config, use_s3=use_s3,<EOL>s3_file=s3_file, preserve_vpc=preserve_vpc)<EOL><DEDENT>else:<EOL><INDENT>create_function(cfg, path_to_zip_file, use_s3=use_s3, s3_file=s3_file)<EOL><DEDENT>

Deploys a new function via AWS S3.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)


def upload(<EOL>src, requirements=None, local_package=None,<EOL>config_file='<STR_LIT>', profile_name=None,<EOL>):

<EOL>path_to_config_file = os.path.join(src, config_file)<EOL>cfg = read_cfg(path_to_config_file, profile_name)<EOL>path_to_zip_file = build(<EOL>src, config_file=config_file, requirements=requirements,<EOL>local_package=local_package,<EOL>)<EOL>upload_s3(cfg, path_to_zip_file)<EOL>

Uploads a new function to AWS S3.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)


def invoke(<EOL>src, event_file='<STR_LIT>',<EOL>config_file='<STR_LIT>', profile_name=None,<EOL>verbose=False,<EOL>):

<EOL>path_to_config_file = os.path.join(src, config_file)<EOL>cfg = read_cfg(path_to_config_file, profile_name)<EOL>if profile_name:<EOL><INDENT>os.environ['<STR_LIT>'] = profile_name<EOL><DEDENT>env_vars = cfg.get('<STR_LIT>')<EOL>if env_vars:<EOL><INDENT>for key, value in env_vars.items():<EOL><INDENT>os.environ[key] = get_environment_variable_value(value)<EOL><DEDENT><DEDENT>path_to_event_file = os.path.join(src, event_file)<EOL>event = read(path_to_event_file, loader=json.loads)<EOL>try:<EOL><INDENT>sys.path.index(src)<EOL><DEDENT>except ValueError:<EOL><INDENT>sys.path.append(src)<EOL><DEDENT>handler = cfg.get('<STR_LIT>')<EOL>fn = get_callable_handler_function(src, handler)<EOL>timeout = cfg.get('<STR_LIT>')<EOL>if timeout:<EOL><INDENT>context = LambdaContext(cfg.get('<STR_LIT>'),timeout)<EOL><DEDENT>else:<EOL><INDENT>context = LambdaContext(cfg.get('<STR_LIT>'))<EOL><DEDENT>start = time.time()<EOL>results = fn(event, context)<EOL>end = time.time()<EOL>print('<STR_LIT>'.format(results))<EOL>if verbose:<EOL><INDENT>print('<STR_LIT>'<EOL>'<STR_LIT>'.format(end - start, cfg.get('<STR_LIT>', <NUM_LIT:15>)))<EOL><DEDENT>

Simulates a call to your function.

    :param str src:
        The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str alt_event:
        An optional argument to override which event file to use.
    :param bool verbose:
        Whether to print out verbose details.


templates_path = os.path.join(<EOL>os.path.dirname(os.path.abspath(__file__)), '<STR_LIT>',<EOL>)<EOL>for filename in os.listdir(templates_path):<EOL><INDENT>if (minimal and filename == '<STR_LIT>') or filename.endswith('<STR_LIT>'):<EOL><INDENT>continue<EOL><DEDENT>dest_path = os.path.join(templates_path, filename)<EOL>if not os.path.isdir(dest_path):<EOL><INDENT>copy(dest_path, src)<EOL><DEDENT><DEDENT>

Copies template files to a given directory.

    :param str src:
        The path to output the template lambda project files.
    :param bool minimal:
        Minimal possible template files (excludes event.json).


def build(<EOL>src, requirements=None, local_package=None,<EOL>config_file='<STR_LIT>', profile_name=None,<EOL>):

<EOL>path_to_config_file = os.path.join(src, config_file)<EOL>cfg = read_cfg(path_to_config_file, profile_name)<EOL>dist_directory = cfg.get('<STR_LIT>', '<STR_LIT>')<EOL>path_to_dist = os.path.join(src, dist_directory)<EOL>mkdir(path_to_dist)<EOL>function_name = cfg.get('<STR_LIT>')<EOL>output_filename = '<STR_LIT>'.format(timestamp(), function_name)<EOL>path_to_temp = mkdtemp(prefix='<STR_LIT>')<EOL>pip_install_to_target(<EOL>path_to_temp,<EOL>requirements=requirements,<EOL>local_package=local_package,<EOL>)<EOL>if '<STR_LIT>' in os.listdir(path_to_temp):<EOL><INDENT>print(<EOL>'<STR_LIT>'<EOL>'<STR_LIT>',<EOL>)<EOL>with open(os.path.join(path_to_temp, '<STR_LIT>'), '<STR_LIT:wb>'):<EOL><INDENT>pass<EOL><DEDENT><DEDENT>output_filename = (<EOL>'<STR_LIT>'.format(output_filename)<EOL>if not output_filename.endswith('<STR_LIT>')<EOL>else output_filename<EOL>)<EOL>build_config = defaultdict(**cfg.get('<STR_LIT>', {}))<EOL>build_source_directories = build_config.get('<STR_LIT>', '<STR_LIT>')<EOL>build_source_directories = (<EOL>build_source_directories<EOL>if build_source_directories is not None<EOL>else '<STR_LIT>'<EOL>)<EOL>source_directories = [<EOL>d.strip() for d in build_source_directories.split('<STR_LIT:U+002C>')<EOL>]<EOL>files = []<EOL>for filename in os.listdir(src):<EOL><INDENT>if os.path.isfile(filename):<EOL><INDENT>if filename == '<STR_LIT>':<EOL><INDENT>continue<EOL><DEDENT>if filename == config_file:<EOL><INDENT>continue<EOL><DEDENT>print('<STR_LIT>' % filename)<EOL>files.append(os.path.join(src, filename))<EOL><DEDENT>elif os.path.isdir(filename) and filename in source_directories:<EOL><INDENT>print('<STR_LIT>' % filename)<EOL>files.append(os.path.join(src, filename))<EOL><DEDENT><DEDENT>os.chdir(path_to_temp)<EOL>for f in files:<EOL><INDENT>if os.path.isfile(f):<EOL><INDENT>_, filename = os.path.split(f)<EOL>copyfile(f, os.path.join(path_to_temp, filename))<EOL>copystat(f, os.path.join(path_to_temp, filename))<EOL><DEDENT>elif os.path.isdir(f):<EOL><INDENT>destination_folder = os.path.join(path_to_temp, f[len(src) + <NUM_LIT:1>:])<EOL>copytree(f, destination_folder)<EOL><DEDENT><DEDENT>path_to_zip_file = archive('<STR_LIT>', path_to_dist, output_filename)<EOL>return path_to_zip_file<EOL>

Builds the file bundle.

    :param str src:
       The path to your Lambda ready project (folder must contain a valid
        config.yaml and handler module (e.g.: service.py).
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)


<EOL>os.chdir(src)<EOL>module_name, function_name = handler.split('<STR_LIT:.>')<EOL>filename = get_handler_filename(handler)<EOL>path_to_module_file = os.path.join(src, filename)<EOL>module = load_source(module_name, path_to_module_file)<EOL>return getattr(module, function_name)<EOL>

Tranlate a string of the form "module.function" into a callable
    function.

    :param str src:
      The path to your Lambda project containing a valid handler file.
    :param str handler:
      A dot delimited string representing the `<module>.<function name>`.


module_name, _ = handler.split('<STR_LIT:.>')<EOL>return '<STR_LIT>'.format(module_name)<EOL>

Shortcut to get the filename from the handler string.

    :param str handler:
      A dot delimited string representing the `<module>.<function name>`.


def _filter_blacklist(package):<EOL><INDENT>blacklist = ['<STR_LIT>', '<STR_LIT:#>', '<STR_LIT>', '<STR_LIT>']<EOL>return all(package.startswith(entry) is False for entry in blacklist)<EOL><DEDENT>filtered_packages = filter(_filter_blacklist, packages)<EOL>for package in filtered_packages:<EOL><INDENT>if package.startswith('<STR_LIT>'):<EOL><INDENT>package = package.replace('<STR_LIT>', '<STR_LIT>')<EOL><DEDENT>print('<STR_LIT>'.format(package=package))<EOL>subprocess.check_call([sys.executable, '<STR_LIT>', '<STR_LIT>', '<STR_LIT>', package, '<STR_LIT>', path, '<STR_LIT>'])<EOL><DEDENT>print ('<STR_LIT>'.format(directory=os.listdir(path)))<EOL>

Install all packages listed to the target directory.

    Ignores any package that includes Python itself and python-lambda as well
    since its only needed for deploying and not running the code

    :param str path:
        Path to copy installed pip packages to.
    :param list packages:
        A list of packages to be installed via pip.


def pip_install_to_target(path, requirements=None, local_package=None):

packages = []<EOL>if not requirements:<EOL><INDENT>print('<STR_LIT>')<EOL>pkgStr = subprocess.check_output([sys.executable, '<STR_LIT>', '<STR_LIT>', '<STR_LIT>'])<EOL>packages.extend(pkgStr.decode('<STR_LIT:utf-8>').splitlines())<EOL><DEDENT>else:<EOL><INDENT>if os.path.exists(requirements):<EOL><INDENT>print('<STR_LIT>')<EOL>data = read(requirements)<EOL>packages.extend(data.splitlines())<EOL><DEDENT><DEDENT>if not packages:<EOL><INDENT>print('<STR_LIT>')<EOL><DEDENT>if local_package is not None:<EOL><INDENT>if not isinstance(local_package, (list, tuple)):<EOL><INDENT>local_package = [local_package]<EOL><DEDENT>for l_package in local_package:<EOL><INDENT>packages.append(l_package)<EOL><DEDENT><DEDENT>_install_packages(path, packages)<EOL>

For a given active virtualenv, gather all installed pip packages then
    copy (re-install) them to the path provided.

    :param str path:
        Path to copy installed pip packages to.
    :param str requirements:
        If set, only the packages in the supplied requirements file are
        installed.
        If not set then installs all packages found via pip freeze.
    :param str local_package:
        The path to a local package with should be included in the deploy as
        well (and/or is not available on PyPi)


def get_role_name(region, account_id, role):

prefix = ARN_PREFIXES.get(region, '<STR_LIT>')<EOL>return '<STR_LIT>'.format(prefix, account_id, role)<EOL>

Shortcut to insert the `account_id` and `role` into the iam string.


def get_account_id(<EOL>profile_name, aws_access_key_id, aws_secret_access_key,<EOL>region=None,<EOL>):

client = get_client(<EOL>'<STR_LIT>', profile_name, aws_access_key_id, aws_secret_access_key,<EOL>region,<EOL>)<EOL>return client.get_caller_identity().get('<STR_LIT>')<EOL>


def get_client(<EOL>client, profile_name, aws_access_key_id, aws_secret_access_key,<EOL>region=None,<EOL>):

Shortcut for getting an initialized instance of the boto3 client.


def create_function(cfg, path_to_zip_file, use_s3=False, s3_file=None):

print('<STR_LIT>')<EOL>byte_stream = read(path_to_zip_file, binary_file=True)<EOL>profile_name = cfg.get('<STR_LIT>')<EOL>aws_access_key_id = cfg.get('<STR_LIT>')<EOL>aws_secret_access_key = cfg.get('<STR_LIT>')<EOL>account_id = get_account_id(<EOL>profile_name, aws_access_key_id, aws_secret_access_key, cfg.get(<EOL>'<STR_LIT>',<EOL>),<EOL>)<EOL>role = get_role_name(<EOL>cfg.get('<STR_LIT>'), account_id,<EOL>cfg.get('<STR_LIT>', '<STR_LIT>'),<EOL>)<EOL>client = get_client(<EOL>'<STR_LIT>', profile_name, aws_access_key_id, aws_secret_access_key,<EOL>cfg.get('<STR_LIT>'),<EOL>)<EOL>buck_name = (<EOL>os.environ.get('<STR_LIT>') or cfg.get('<STR_LIT>')<EOL>)<EOL>func_name = (<EOL>os.environ.get('<STR_LIT>') or cfg.get('<STR_LIT>')<EOL>)<EOL>print('<STR_LIT>'.format(func_name))<EOL>if use_s3:<EOL><INDENT>kwargs = {<EOL>'<STR_LIT>': func_name,<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', '<STR_LIT>'),<EOL>'<STR_LIT>': role,<EOL>'<STR_LIT>': cfg.get('<STR_LIT>'),<EOL>'<STR_LIT>': {<EOL>'<STR_LIT>': '<STR_LIT:{}>'.format(buck_name),<EOL>'<STR_LIT>': '<STR_LIT:{}>'.format(s3_file),<EOL>},<EOL>'<STR_LIT>': cfg.get('<STR_LIT:description>', '<STR_LIT>'),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', <NUM_LIT:15>),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', <NUM_LIT>),<EOL>'<STR_LIT>': {<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>},<EOL>'<STR_LIT>': True,<EOL>}<EOL><DEDENT>else:<EOL><INDENT>kwargs = {<EOL>'<STR_LIT>': func_name,<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', '<STR_LIT>'),<EOL>'<STR_LIT>': role,<EOL>'<STR_LIT>': cfg.get('<STR_LIT>'),<EOL>'<STR_LIT>': {'<STR_LIT>': byte_stream},<EOL>'<STR_LIT>': cfg.get('<STR_LIT:description>', '<STR_LIT>'),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', <NUM_LIT:15>),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', <NUM_LIT>),<EOL>'<STR_LIT>': {<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>},<EOL>'<STR_LIT>': True,<EOL>}<EOL><DEDENT>if '<STR_LIT>' in cfg:<EOL><INDENT>kwargs.update(<EOL>Tags={<EOL>key: str(value)<EOL>for key, value in cfg.get('<STR_LIT>').items()<EOL>}<EOL>)<EOL><DEDENT>if '<STR_LIT>' in cfg:<EOL><INDENT>kwargs.update(<EOL>Environment={<EOL>'<STR_LIT>': {<EOL>key: get_environment_variable_value(value)<EOL>for key, value<EOL>in cfg.get('<STR_LIT>').items()<EOL>},<EOL>},<EOL>)<EOL><DEDENT>client.create_function(**kwargs)<EOL>concurrency = get_concurrency(cfg)<EOL>if concurrency > <NUM_LIT:0>:<EOL><INDENT>client.put_function_concurrency(FunctionName=func_name, ReservedConcurrentExecutions=concurrency)<EOL><DEDENT>

Register and upload a function to AWS Lambda.


def update_function(<EOL>cfg, path_to_zip_file, existing_cfg, use_s3=False, s3_file=None, preserve_vpc=False<EOL>):

print('<STR_LIT>')<EOL>byte_stream = read(path_to_zip_file, binary_file=True)<EOL>profile_name = cfg.get('<STR_LIT>')<EOL>aws_access_key_id = cfg.get('<STR_LIT>')<EOL>aws_secret_access_key = cfg.get('<STR_LIT>')<EOL>account_id = get_account_id(<EOL>profile_name, aws_access_key_id, aws_secret_access_key, cfg.get(<EOL>'<STR_LIT>',<EOL>),<EOL>)<EOL>role = get_role_name(<EOL>cfg.get('<STR_LIT>'), account_id,<EOL>cfg.get('<STR_LIT>', '<STR_LIT>'),<EOL>)<EOL>client = get_client(<EOL>'<STR_LIT>', profile_name, aws_access_key_id, aws_secret_access_key,<EOL>cfg.get('<STR_LIT>'),<EOL>)<EOL>buck_name = (<EOL>os.environ.get('<STR_LIT>') or cfg.get('<STR_LIT>')<EOL>)<EOL>if use_s3:<EOL><INDENT>client.update_function_code(<EOL>FunctionName=cfg.get('<STR_LIT>'),<EOL>S3Bucket='<STR_LIT:{}>'.format(buck_name),<EOL>S3Key='<STR_LIT:{}>'.format(s3_file),<EOL>Publish=True,<EOL>)<EOL><DEDENT>else:<EOL><INDENT>client.update_function_code(<EOL>FunctionName=cfg.get('<STR_LIT>'),<EOL>ZipFile=byte_stream,<EOL>Publish=True,<EOL>)<EOL><DEDENT>kwargs = {<EOL>'<STR_LIT>': cfg.get('<STR_LIT>'),<EOL>'<STR_LIT>': role,<EOL>'<STR_LIT>': cfg.get('<STR_LIT>'),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>'),<EOL>'<STR_LIT>': cfg.get('<STR_LIT:description>', '<STR_LIT>'),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', <NUM_LIT:15>),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', <NUM_LIT>),<EOL>}<EOL>if preserve_vpc:<EOL><INDENT>kwargs['<STR_LIT>'] = existing_cfg.get('<STR_LIT>', {}).get('<STR_LIT>')<EOL>if kwargs['<STR_LIT>'] is None:<EOL><INDENT>kwargs['<STR_LIT>'] = {<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>}<EOL><DEDENT>else:<EOL><INDENT>del kwargs['<STR_LIT>']['<STR_LIT>']<EOL><DEDENT><DEDENT>else:<EOL><INDENT>kwargs['<STR_LIT>'] = {<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>'<STR_LIT>': cfg.get('<STR_LIT>', []),<EOL>}<EOL><DEDENT>if '<STR_LIT>' in cfg:<EOL><INDENT>kwargs.update(<EOL>Environment={<EOL>'<STR_LIT>': {<EOL>key: str(get_environment_variable_value(value))<EOL>for key, value<EOL>in cfg.get('<STR_LIT>').items()<EOL>},<EOL>},<EOL>)<EOL><DEDENT>ret = client.update_function_configuration(**kwargs)<EOL>concurrency = get_concurrency(cfg)<EOL>if concurrency > <NUM_LIT:0>:<EOL><INDENT>client.put_function_concurrency(FunctionName=cfg.get('<STR_LIT>'), ReservedConcurrentExecutions=concurrency)<EOL><DEDENT>elif '<STR_LIT>' in existing_cfg:<EOL><INDENT>client.delete_function_concurrency(FunctionName=cfg.get('<STR_LIT>'))<EOL><DEDENT>if '<STR_LIT>' in cfg:<EOL><INDENT>tags = {<EOL>key: str(value)<EOL>for key, value in cfg.get('<STR_LIT>').items()<EOL>}<EOL>if tags != existing_cfg.get('<STR_LIT>'):<EOL><INDENT>if existing_cfg.get('<STR_LIT>'):<EOL><INDENT>client.untag_resource(Resource=ret['<STR_LIT>'],<EOL>TagKeys=list(existing_cfg['<STR_LIT>'].keys()))<EOL><DEDENT>client.tag_resource(Resource=ret['<STR_LIT>'], Tags=tags)<EOL><DEDENT><DEDENT>

Updates the code of an existing Lambda function


def upload_s3(cfg, path_to_zip_file, *use_s3):

print('<STR_LIT>')<EOL>profile_name = cfg.get('<STR_LIT>')<EOL>aws_access_key_id = cfg.get('<STR_LIT>')<EOL>aws_secret_access_key = cfg.get('<STR_LIT>')<EOL>client = get_client(<EOL>'<STR_LIT>', profile_name, aws_access_key_id, aws_secret_access_key,<EOL>cfg.get('<STR_LIT>'),<EOL>)<EOL>byte_stream = b'<STR_LIT>'<EOL>with open(path_to_zip_file, mode='<STR_LIT:rb>') as fh:<EOL><INDENT>byte_stream = fh.read()<EOL><DEDENT>s3_key_prefix = cfg.get('<STR_LIT>', '<STR_LIT>')<EOL>checksum = hashlib.new('<STR_LIT>', byte_stream).hexdigest()<EOL>timestamp = str(time.time())<EOL>filename = '<STR_LIT>'.format(<EOL>prefix=s3_key_prefix, checksum=checksum, ts=timestamp,<EOL>)<EOL>buck_name = (<EOL>os.environ.get('<STR_LIT>') or cfg.get('<STR_LIT>')<EOL>)<EOL>func_name = (<EOL>os.environ.get('<STR_LIT>') or cfg.get('<STR_LIT>')<EOL>)<EOL>kwargs = {<EOL>'<STR_LIT>': '<STR_LIT:{}>'.format(buck_name),<EOL>'<STR_LIT>': '<STR_LIT:{}>'.format(filename),<EOL>'<STR_LIT>': byte_stream,<EOL>}<EOL>client.put_object(**kwargs)<EOL>print('<STR_LIT>'.format(func_name, buck_name))<EOL>if use_s3:<EOL><INDENT>return filename<EOL><DEDENT>


function_name = cfg.get('<STR_LIT>')<EOL>profile_name = cfg.get('<STR_LIT>')<EOL>aws_access_key_id = cfg.get('<STR_LIT>')<EOL>aws_secret_access_key = cfg.get('<STR_LIT>')<EOL>client = get_client(<EOL>'<STR_LIT>', profile_name, aws_access_key_id, aws_secret_access_key,<EOL>cfg.get('<STR_LIT>'),<EOL>)<EOL>try:<EOL><INDENT>return client.get_function(FunctionName=function_name)<EOL><DEDENT>except client.exceptions.ResourceNotFoundException as e:<EOL><INDENT>if '<STR_LIT>' in str(e):<EOL><INDENT>return False<EOL><DEDENT><DEDENT>

Check whether a function exists or not and return its config


concurrency = int(cfg.get('<STR_LIT>', <NUM_LIT:0>))<EOL>return max(<NUM_LIT:0>, concurrency)<EOL>

Return the Reserved Concurrent Executions if present in the config


try:<EOL><INDENT>logging.config.dictConfig(<EOL>create_log_config(verbose=verbose, quiet=quiet)<EOL>)<EOL><DEDENT>except ValueError as e:<EOL><INDENT>click.secho(e.args[<NUM_LIT:0>], fg="<STR_LIT>")<EOL>raise SystemExit(<NUM_LIT:1>)<EOL><DEDENT>if icon:<EOL><INDENT>icon_data = icon.read()<EOL>if not icon_data.startswith(PNG_HEADER):<EOL><INDENT>log.error(<EOL>'<STR_LIT>'.format(<EOL>click.format_filename(icon.name)<EOL>)<EOL>)<EOL>raise SystemExit(<NUM_LIT:1>)<EOL><DEDENT><DEDENT>else:<EOL><INDENT>icon_data = None<EOL><DEDENT>source, dest, name = setup_paths(<EOL>source,<EOL>destination,<EOL>name=name,<EOL>add_to_global=add_to_global,<EOL>force=force,<EOL>)<EOL>if parser is None:<EOL><INDENT>parser = parsers.get_doctype(source)<EOL>if parser is None:<EOL><INDENT>log.error(<EOL>'<STR_LIT>'.format(<EOL>click.format_filename(source)<EOL>)<EOL>)<EOL>raise SystemExit(errno.EINVAL)<EOL><DEDENT><DEDENT>docset = prepare_docset(<EOL>source, dest, name, index_page, enable_js, online_redirect_url<EOL>)<EOL>doc_parser = parser(doc_path=docset.docs)<EOL>log.info(<EOL>(<EOL>"<STR_LIT>"<EOL>+ click.style("<STR_LIT>", bold=True)<EOL>+ '<STR_LIT>'<EOL>).format(<EOL>parser_name=parser.name,<EOL>src=click.format_filename(source, shorten=True),<EOL>dst=click.format_filename(dest),<EOL>)<EOL>)<EOL>with docset.db_conn:<EOL><INDENT>log.info("<STR_LIT>")<EOL>toc = patch_anchors(doc_parser, show_progressbar=not quiet)<EOL>for entry in doc_parser.parse():<EOL><INDENT>docset.db_conn.execute(<EOL>"<STR_LIT>",<EOL>entry.as_tuple(),<EOL>)<EOL>toc.send(entry)<EOL><DEDENT>count = docset.db_conn.execute(<EOL>"<STR_LIT>"<EOL>).fetchone()[<NUM_LIT:0>]<EOL>log.info(<EOL>(<EOL>"<STR_LIT>"<EOL>+ click.style("<STR_LIT>", fg="<STR_LIT>" if count > <NUM_LIT:0> else "<STR_LIT>")<EOL>+ "<STR_LIT>"<EOL>).format(count=count)<EOL>)<EOL>toc.close()<EOL><DEDENT>if icon_data:<EOL><INDENT>add_icon(icon_data, dest)<EOL><DEDENT>if add_to_dash or add_to_global:<EOL><INDENT>log.info("<STR_LIT>")<EOL>os.system('<STR_LIT>'.format(dest))<EOL><DEDENT>

Convert docs from SOURCE to Dash.app's docset format.