def WelchPeriodogram(data, NFFT=None,  sampling=<NUM_LIT:1.>, **kargs):

from pylab import psd<EOL>spectrum = Spectrum(data, sampling=<NUM_LIT:1.>)<EOL>P = psd(data, NFFT, Fs=sampling, **kargs)<EOL>spectrum.psd = P[<NUM_LIT:0>]<EOL>return P, spectrum<EOL>

r"""Simple periodogram wrapper of numpy.psd function.

    :param A: the input data
    :param int NFFT: total length of the final data sets (padded
        with zero if needed; default is 4096)
    :param str window:

    :Technical documentation:

    When we calculate the periodogram of a set of data we get an estimation
    of the spectral density. In fact as we use a Fourier transform and a
    truncated segments the spectrum is the convolution of the data with a
    rectangular window which Fourier transform is

    .. math::

        W(s)= \frac{1}{N^2} \left[ \frac{\sin(\pi s)}{\sin(\pi s/N)} \right]^2

    Thus oscillations and sidelobes appears around the main frequency. One aim of t he tapering is to reduced this effects. We multiply data by a window whose  sidelobes are much smaller than the main lobe. Classical window is hanning window.  But other windows are available. However we must take into account this energy and divide the spectrum by energy of taper used. Thus periodogram becomes :

    .. math::

        D_k \equiv \sum_{j=0}^{N-1}c_jw_j \; e^{2\pi ijk/N}  \qquad k=0,...,N-1

    .. math::

        P(0)=P(f_0)=\frac{1}{2\pi W_{ss}}\arrowvert{D_0}\arrowvert^2

    .. math::

        P(f_k)=\frac{1}{2\pi W_{ss}} \left[\arrowvert{D_k}\arrowvert^2+\arrowvert{D_{N-k}}\arrowvert^2\right]        \qquad k=0,1,...,     \left( \frac{1}{2}-1 \right)

    .. math::

        P(f_c)=P(f_{N/2})= \frac{1}{2\pi W_{ss}} \arrowvert{D_{N/2}}\arrowvert^2

    with

    .. math::

        {W_{ss}} \equiv N\sum_{j=0}^{N-1}w_j^2

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import WelchPeriodogram, marple_data
        psd = WelchPeriodogram(marple_data, 256)


if Q <= <NUM_LIT:0> or Q >= M:<EOL><INDENT>raise ValueError('<STR_LIT>')<EOL><DEDENT>a, rho, _c = yulewalker.aryule(X, M, '<STR_LIT>')   <EOL>a = np.insert(a, <NUM_LIT:0>, <NUM_LIT:1>)<EOL>ma_params, _p, _c = yulewalker.aryule(a, Q, '<STR_LIT>')    <EOL>return ma_params, rho<EOL>

Moving average estimator.

    This program provides an estimate of the moving average parameters
    and driving noise variance for a data sequence based on a
    long AR model and a least squares fit.

    :param array X: The input data array
    :param int Q: Desired MA model order (must be >0 and <M)
    :param int M: Order of "long" AR model (suggest at least 2*Q )

    :return:
        * MA    - Array of Q complex MA parameter estimates
        * RHO   - Real scalar of white noise variance estimate

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import arma2psd, ma, marple_data
        import pylab

        # Estimate 15 Ma parameters
        b, rho = ma(marple_data, 15, 30)
        # Create the PSD from those MA parameters
        psd = arma2psd(B=b, rho=rho, sides='centerdc')
        # and finally plot the PSD
        pylab.plot(pylab.linspace(-0.5, 0.5, 4096), 10 * pylab.log10(psd/max(psd)))
        pylab.axis([-0.5, 0.5, -30, 0])

    :reference: [Marple]_


R = CORRELATION(X, maxlags=lag, norm='<STR_LIT>')<EOL>R0 = R[<NUM_LIT:0>]<EOL>MPQ = lag - Q + P<EOL>N = len(X)<EOL>Y = np.zeros(N-P, dtype=complex)<EOL>for K in range(<NUM_LIT:0>, MPQ):<EOL><INDENT>KPQ = K + Q - P+<NUM_LIT:1><EOL>if KPQ < <NUM_LIT:0>:<EOL><INDENT>Y[K] = R[-KPQ].conjugate()<EOL><DEDENT>if KPQ == <NUM_LIT:0>:<EOL><INDENT>Y[K] = R0<EOL><DEDENT>if KPQ > <NUM_LIT:0>:<EOL><INDENT>Y[K] = R[KPQ]<EOL><DEDENT><DEDENT>Y.resize(lag)<EOL>if P <= <NUM_LIT:4>:<EOL><INDENT>res = arcovar_marple(Y.copy(), P)    <EOL>ar_params = res[<NUM_LIT:0>]<EOL><DEDENT>else:<EOL><INDENT>res = arcovar(Y.copy(), P)    <EOL>ar_params = res[<NUM_LIT:0>]<EOL><DEDENT>Y.resize(N-P)<EOL>for k in range(P, N):<EOL><INDENT>SUM = X[k]<EOL>for j in range(<NUM_LIT:0>, P):<EOL><INDENT>SUM = SUM + ar_params[j] * X[k-j-<NUM_LIT:1>]   <EOL><DEDENT>Y[k-P] = SUM<EOL><DEDENT>ma_params, rho = ma(Y, Q, <NUM_LIT:2>*Q)     <EOL>return ar_params, ma_params, rho<EOL>

Autoregressive and moving average estimators.

    This function provides an estimate of the autoregressive
    parameters, the moving average parameters, and the driving
    white noise variance of  an ARMA(P,Q) for a complex or real data sequence.

    The parameters are estimated using three steps:

        * Estimate the AR parameters from the original data based on a least
          squares modified Yule-Walker technique,
        * Produce a residual time sequence by filtering the original data
          with a filter based on the AR parameters,
        * Estimate the MA parameters from the residual time sequence.

    :param array X: Array of data samples (length N)
    :param int P: Desired number of AR parameters
    :param int Q: Desired number of MA parameters
    :param int lag: Maximum lag to use for autocorrelation estimates

    :return:
        * A     - Array of complex P AR parameter estimates
        * B     - Array of complex Q MA parameter estimates
        * RHO   - White noise variance estimate

    .. note::
      *  lag must be >= Q (MA order)

    **dependencies**:
        * :meth:`spectrum.correlation.CORRELATION`
        * :meth:`spectrum.covar.arcovar`
        * :meth:`spectrum.arma.ma`

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import arma_estimate, arma2psd, marple_data
        import pylab

        a,b, rho = arma_estimate(marple_data, 15, 15, 30)
        psd = arma2psd(A=a, B=b, rho=rho, sides='centerdc', norm=True)
        pylab.plot(10 * pylab.log10(psd))
        pylab.ylim([-50,0])

    :reference: [Marple]_


def __init__(self, data, P, Q, lag, NFFT=None, sampling=<NUM_LIT:1.>,<EOL>scale_by_freq=False):

super(parma, self).__init__(data, ma_order=Q, ar_order=P, lag=lag,<EOL>NFFT=NFFT, sampling=sampling,<EOL>scale_by_freq=scale_by_freq)<EOL>self.lag = lag<EOL>

**Constructor:**

        For a detailed description of the parameters, see :func:`arma_estimate`.

        :param array data:     input data (list or numpy.array)
        :param int P:
        :param int Q:
        :param int lag:
        :param int NFFT: total length of the final data sets (padded with
            zero if needed; default is 4096)
        :param float sampling: sampling frequency of the input :attr:`data`.


def arma2psd(A=None, B=None, rho=<NUM_LIT:1.>, T=<NUM_LIT:1.>, NFFT=<NUM_LIT>, sides='<STR_LIT:default>',<EOL>norm=False):

if NFFT is None:<EOL><INDENT>NFFT = <NUM_LIT><EOL><DEDENT>if A is None and B is None:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>psd = np.zeros(NFFT, dtype=complex)<EOL>if A is not None:<EOL><INDENT>ip = len(A)<EOL>den = np.zeros(NFFT, dtype=complex)<EOL>den[<NUM_LIT:0>] = <NUM_LIT:1.>+<NUM_LIT><EOL>for k in range(<NUM_LIT:0>, ip):<EOL><INDENT>den[k+<NUM_LIT:1>] = A[k]<EOL><DEDENT>denf = fft(den, NFFT)<EOL><DEDENT>if B is not None:<EOL><INDENT>iq = len(B)<EOL>num = np.zeros(NFFT, dtype=complex)<EOL>num[<NUM_LIT:0>] = <NUM_LIT:1.>+<NUM_LIT><EOL>for k in range(<NUM_LIT:0>, iq):<EOL><INDENT>num[k+<NUM_LIT:1>] = B[k]<EOL><DEDENT>numf = fft(num, NFFT)<EOL><DEDENT>if A is not None and B is not None:<EOL><INDENT>psd = rho / T * abs(numf)**<NUM_LIT> / abs(denf)**<NUM_LIT><EOL><DEDENT>elif A is not None:<EOL><INDENT>psd = rho / T / abs(denf)**<NUM_LIT><EOL><DEDENT>elif B is not None:<EOL><INDENT>psd = rho / T * abs(numf)**<NUM_LIT><EOL><DEDENT>psd = np.real(psd)<EOL>if sides != '<STR_LIT:default>':<EOL><INDENT>from . import tools<EOL>assert sides in ['<STR_LIT>']<EOL>if sides == '<STR_LIT>':<EOL><INDENT>psd = tools.twosided_2_centerdc(psd)<EOL><DEDENT><DEDENT>if norm == True:<EOL><INDENT>psd /= max(psd)<EOL><DEDENT>return psd<EOL>

r"""Computes power spectral density given ARMA values.

    This function computes the power spectral density values
    given the ARMA parameters of an ARMA model. It assumes that
    the driving sequence is a white noise process of zero mean and
    variance :math:`\rho_w`. The sampling frequency and noise variance are
    used to scale the PSD output, which length is set by the user with the
    `NFFT` parameter.

    :param array A:   Array of AR parameters (complex or real)
    :param array B:   Array of MA parameters (complex or real)
    :param float rho: White noise variance to scale the returned PSD
    :param float T:   Sample interval in seconds to scale the returned PSD
    :param int NFFT:  Final size of the PSD
    :param str sides: Default PSD is two-sided, but sides can be set to centerdc.

    .. warning:: By convention, the AR or MA arrays does not contain the
        A0=1 value.

    If :attr:`B` is None, the model is a pure AR model. If :attr:`A` is None,
    the model is a pure MA model.

    :return: two-sided PSD

    .. rubric:: Details:

    AR case: the power spectral density is:

    .. math:: P_{ARMA}(f) = T \rho_w \left|\frac{B(f)}{A(f)}\right|^2

    where:

    .. math:: A(f) = 1 + \sum_{k=1}^q b(k) e^{-j2\pi fkT}
    .. math:: B(f) = 1 + \sum_{k=1}^p a(k) e^{-j2\pi fkT}

    .. rubric:: **Example:**

    .. plot::
        :width: 80%
        :include-source:

        import spectrum.arma
        from pylab import plot, log10, legend
        plot(10*log10(spectrum.arma.arma2psd([1,0.5],[0.5,0.5])), label='ARMA(2,2)')
        plot(10*log10(spectrum.arma.arma2psd([1,0.5],None)), label='AR(2)')
        plot(10*log10(spectrum.arma.arma2psd(None,[0.5,0.5])), label='MA(2)')
        legend()

    :References: [Marple]_


m = len(x)<EOL>if N is None:<EOL><INDENT>N = m - <NUM_LIT:1> <EOL><DEDENT>elif N > m-<NUM_LIT:1>:<EOL><INDENT>x.resize(N+<NUM_LIT:1>)<EOL><DEDENT>X = fft(x, <NUM_LIT:2>**nextpow2(<NUM_LIT>*len(x)-<NUM_LIT:1>))<EOL>R = real(ifft(abs(X)**<NUM_LIT:2>))<EOL>R = R/(m-<NUM_LIT:1.>) <EOL>a, e, ref = LEVINSON(R, N)<EOL>return a, e<EOL>

Linear Predictor Coefficients.

    :param x:
    :param int N: default is length(X) - 1

    :Details:

    Finds the coefficients :math:`A=(1, a(2), \dots a(N+1))`, of an Nth order
    forward linear predictor that predicts the current value value of the
    real-valued time series x based on past samples:

    .. math:: \hat{x}(n) = -a(2)*x(n-1) - a(3)*x(n-2) - ... - a(N+1)*x(n-N)

    such that the sum of the squares of the errors

    .. math:: err(n) = X(n) - Xp(n)

    is minimized. This function  uses the Levinson-Durbin recursion to
    solve the normal equations that arise from the least-squares formulation.

    .. seealso:: :func:`levinson`, :func:`aryule`, :func:`prony`, :func:`stmcb`

    .. todo:: matrix case, references

    :Example:

    ::

        from scipy.signal import lfilter
        noise = randn(50000,1);  % Normalized white Gaussian noise
        x = filter([1], [1 1/2 1/3 1/4], noise)
        x = x[45904:50000]
        x.reshape(4096, 1)
        x = x[0]

    Compute the predictor coefficients, estimated signal, prediction error, and autocorrelation sequence of the prediction error:

    1.00000 + 0.00000i   0.51711 - 0.00000i   0.33908 - 0.00000i   0.24410 - 0.00000i

    ::

        a = lpc(x, 3)
        est_x = lfilter([0 -a(2:end)],1,x);    % Estimated signal
        e = x - est_x;                        % Prediction error
        [acs,lags] = xcorr(e,'coeff');   % ACS of prediction error


def _get_signal_space(S, NP, verbose=False, threshold=None, NSIG=None,<EOL>criteria='<STR_LIT>'):

from .criteria import aic_eigen, mdl_eigen<EOL>if NSIG is None:<EOL><INDENT>if threshold is None:<EOL><INDENT>logging.debug('<STR_LIT>')<EOL>if criteria == '<STR_LIT>':<EOL><INDENT>aic = aic_eigen(S, NP*<NUM_LIT:2>)<EOL><DEDENT>elif criteria == '<STR_LIT>':<EOL><INDENT>aic = mdl_eigen(S, NP*<NUM_LIT:2>)<EOL><DEDENT>NSIG = np.argmin(aic) + <NUM_LIT:1><EOL>logging.debug('<STR_LIT>', NSIG, '<STR_LIT>')<EOL><DEDENT>else:<EOL><INDENT>logging.debug('<STR_LIT>')<EOL>m = threshold * min(S)<EOL>new_s = S[np.where(S>m)]<EOL>NSIG = len(new_s)<EOL>logging.debug('<STR_LIT>', NSIG)<EOL>if NSIG == <NUM_LIT:0>:<EOL><INDENT>NSIG = <NUM_LIT:1><EOL><DEDENT><DEDENT><DEDENT>return NSIG<EOL>


def eigen(X, P, NSIG=None, method='<STR_LIT>', threshold=None, NFFT=default_NFFT,<EOL>criteria='<STR_LIT>', verbose=False):

if method not in ['<STR_LIT>', '<STR_LIT>']:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if NSIG != None and threshold != None:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if NSIG is not None:<EOL><INDENT>if NSIG < <NUM_LIT:0>:<EOL><INDENT>raise ValueError('<STR_LIT>')<EOL><DEDENT>if NSIG >= P:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT><DEDENT>N = len(X)<EOL>NP = N - P<EOL>assert <NUM_LIT:2> * NP > P-<NUM_LIT:1>, '<STR_LIT>'<EOL>if NP > <NUM_LIT:100>:<EOL><INDENT>NP = <NUM_LIT:100><EOL><DEDENT>FB = np.zeros((<NUM_LIT:2>*NP, P), dtype=complex)<EOL>Z = np.zeros(NFFT, dtype=complex)<EOL>PSD = np.zeros(NFFT)<EOL>for I in range(<NUM_LIT:0>, NP):<EOL><INDENT>for K in range(<NUM_LIT:0>, P):<EOL><INDENT>FB[I, K] = X[I-K+P-<NUM_LIT:1>]<EOL>FB[I+NP, K] = X[I+K+<NUM_LIT:1>].conjugate()<EOL><DEDENT><DEDENT>_U, S, V = svd (FB)<EOL>V = -V.transpose()<EOL>NSIG  = _get_signal_space(S, <NUM_LIT:2>*NP,<EOL>verbose=verbose, threshold=threshold,<EOL>NSIG=NSIG, criteria=criteria)<EOL>for I in range(NSIG, P):<EOL><INDENT>Z[<NUM_LIT:0>:P] = V[<NUM_LIT:0>:P, I]<EOL>Z[P:NFFT] = <NUM_LIT:0><EOL>Z  = fft(Z, NFFT)<EOL>if method == '<STR_LIT>':<EOL><INDENT>PSD = PSD + abs(Z)**<NUM_LIT><EOL><DEDENT>elif method == '<STR_LIT>' :<EOL><INDENT>PSD = PSD + abs(Z)**<NUM_LIT> / S[I]<EOL><DEDENT><DEDENT>PSD = <NUM_LIT:1.>/PSD<EOL>nby2 = int(NFFT/<NUM_LIT:2>)<EOL>newpsd = np.append(PSD[nby2:<NUM_LIT:0>:-<NUM_LIT:1>], PSD[nby2*<NUM_LIT:2>-<NUM_LIT:1>:nby2-<NUM_LIT:1>:-<NUM_LIT:1>])<EOL>return newpsd, S<EOL>

r"""Pseudo spectrum using eigenvector method (EV or Music)

    This function computes either the Music or EigenValue (EV) noise
    subspace frequency estimator.

    First, an autocorrelation matrix of order `P` is computed from
    the data. Second, this matrix is separated into vector subspaces,
    one a signal subspace and the other a noise
    subspace using a SVD method to obtain the eigen values and vectors.
    From the eigen values :math:`\lambda_i`, and eigen vectors :math:`v_k`,
    the **pseudo spectrum** (see note below) is computed as follows:

    .. math:: P_{ev}(f) = \frac{1}{e^H(f)\left(\sum\limits_{k=M+1}^{p} \frac{1}{\lambda_k}v_kv_k^H\right)e(f)}

    The separation of the noise and signal subspaces requires expertise
    of the signal. However, AIC and MDL criteria may be used to automatically
    perform this task.

    You still need to provide the parameter `P` to indicate the maximum number
    of eigen values to be computed. The criteria will just select a subset
    to estimate the pseudo spectrum (see :func:`~spectrum.criteria.aic_eigen`
    and :func:`~spectrum.criteria.mdl_eigen` for details.

    .. note:: **pseudo spectrum**. func:`eigen` does not compute a PSD estimate.
        Indeed, the method does not preserve the measured process power.

    :param X: Array data samples
    :param int P: maximum number of eigen values to compute. NSIG (if
        specified) must therefore be less than P.
    :param str method: 'music' or 'ev'.
    :param int NSIG: If specified, the signal sub space uses NSIG eigen values.
    :param float threshold: If specified, the signal sub space is made of the
        eigen values larger than :math:`\rm{threshold} \times \lambda_{min}`,
        where :math:`\lambda_{min}` is the minimum eigen values.
    :param int NFFT: total length of the final data sets (padded with zero
        if needed; default is 4096)

    :return:
        * PSD: Array of real frequency estimator values (two sided for
                complex data and one sided for real data)
        * S, the eigen values

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import eigen, marple_data
        from pylab import plot, log10, linspace, legend, axis

        psd, ev = eigen(marple_data, 15, NSIG=11)
        f = linspace(-0.5, 0.5, len(psd))
        plot(f, 10 * log10(psd/max(psd)), label='User defined')

        psd, ev = eigen(marple_data, 15, threshold=2)
        plot(f, 10 * log10(psd/max(psd)), label='threshold method (100)')

        psd, ev = eigen(marple_data, 15)
        plot(f, 10 * log10(psd/max(psd)), label='AIC method (8)')

        legend()
        axis([-0.5, 0.5, -120, 0])

    .. seealso::
        :func:`pev`,
        :func:`pmusic`,
        :func:`~spectrum.criteria.aic_eigen`

    :References: [Marple]_, Chap 13

    .. todo:: for developers:

        * what should be the second argument of the criteria N, N-P, P...?
        * what should be the max value of NP


def music(X, IP, NSIG=None, NFFT=default_NFFT, threshold=None, criteria='<STR_LIT>',<EOL>verbose=False):

return eigen(X, IP, NSIG=NSIG, method='<STR_LIT>', NFFT=NFFT,<EOL>threshold=threshold, criteria=criteria, verbose=verbose)<EOL>

Eigen value pseudo spectrum estimate. See :func:`eigenfre`


def __init__(self, data, IP, NSIG=None, NFFT=None, sampling=<NUM_LIT:1.>,<EOL>threshold=None, criteria="<STR_LIT>", verbose=False, scale_by_freq=False):

super(pmusic, self).__init__(data, ar_order=IP,<EOL>scale_by_freq=scale_by_freq, NFFT=NFFT, sampling=sampling)<EOL>self.NSIG = NSIG<EOL>self.threshold = threshold<EOL>self.criteria = criteria<EOL>self.verbose = verbose<EOL>

**Constructor:**

        For a detailed description of the parameters, see :func:`arma_estimate`.

        :param array data: input data (list or numpy.array)
        :param int P: maximum number of eigen values to compute. NSIG (if
            specified) must therefore be less than P.
        :param int NSIG: If specified, the signal sub space uses NSIG eigen values.
        :param int NFFT: total length of the final data sets (padded with zero if needed; default is 4096)
        :param float sampling: sampling frequency of the input :attr:`data`.


def __init__(self, data, IP, NSIG=None, NFFT=None, sampling=<NUM_LIT:1.>,<EOL>scale_by_freq=False,<EOL>threshold=None, criteria="<STR_LIT>", verbose=False):

super(pev, self).__init__(data, ar_order=IP,<EOL>scale_by_freq=scale_by_freq, NFFT=NFFT, sampling=sampling)<EOL>self.NSIG = NSIG<EOL>self.threshold = threshold<EOL>self.criteria = criteria<EOL>self.verbose = verbose<EOL>

**Constructor:**

        For a detailed description of the parameters, see :func:`arma_estimate`.

        :param array data: input data (list or numpy.array)
        :param int P: maximum number of eigen values to compute. NSIG (if
            specified) must therefore be less than P.
        :param int NSIG: If specified, the signal sub space uses NSIG eigen values.
        :param int NFFT: total length of the final data sets (padded with
            zero if needed; default is 4096)
        :param float sampling: sampling frequency of the input :attr:`data`.


def ev(X, IP, NSIG=None, NFFT=default_NFFT, threshold=None, criteria='<STR_LIT>',<EOL>verbose=False):


return <NUM_LIT>**(xdb/<NUM_LIT>)<EOL>

Convert decibels (dB) to magnitude

    .. doctest::

        >>> from spectrum import db2mag
        >>> db2mag(-20)
        0.1

    .. seealso:: :func:`pow2db`


assert len(data) % <NUM_LIT:2> == <NUM_LIT:0><EOL>N = len(data)<EOL>psd = np.array(data[<NUM_LIT:0>:N//<NUM_LIT:2>+<NUM_LIT:1>]) * <NUM_LIT><EOL>psd[<NUM_LIT:0>] /= <NUM_LIT><EOL>psd[-<NUM_LIT:1>] = data[-<NUM_LIT:1>]<EOL>return psd<EOL>

Convert a one-sided PSD to a twosided PSD

    In order to keep the power in the onesided PSD the same
    as in the twosided version, the onesided values are twice
    as much as in the input data (except for the zero-lag value).

    ::

        >>> twosided_2_onesided([10, 2,3,3,2,8])
        array([ 10.,   4.,   6.,   8.])


return np.fft.fftshift(x)<EOL>

wrapper to numpy.fft.fftshift

    .. doctest::

        >>> from spectrum import fftshift
        >>> x = [100, 2, 3, 4, 5]
        >>> fftshift(x)
        array([  4,   5, 100,   2,   3])


psd = np.concatenate((data[<NUM_LIT:0>:-<NUM_LIT:1>], cshift(data[-<NUM_LIT:1>:<NUM_LIT:0>:-<NUM_LIT:1>], -<NUM_LIT:1>)))/<NUM_LIT><EOL>psd[<NUM_LIT:0>] *= <NUM_LIT><EOL>psd[-<NUM_LIT:1>] *= <NUM_LIT><EOL>return psd<EOL>

Convert a two-sided PSD to a one-sided PSD

    In order to keep the power in the twosided PSD the same
    as in the onesided version, the twosided values are 2 times
    lower than the input data (except for the zero-lag and N-lag
    values).

    ::

        >>> twosided_2_onesided([10, 4, 6, 8])
        array([ 10.,   2.,   3.,   3., 2., 8.])


return <NUM_LIT>**(xdb/<NUM_LIT>)<EOL>

Convert decibels (dB) to power

    .. doctest::

        >>> from spectrum import db2pow
        >>> p = db2pow(-10)
        >>> p
        0.1

    .. seealso:: :func:`pow2db`


def __init__(self, data, order, norm='<STR_LIT>', NFFT=None, sampling=<NUM_LIT:1.>,<EOL>scale_by_freq=True):

super(pyule, self).__init__(data, ar_order=order,  NFFT=NFFT,<EOL>scale_by_freq=scale_by_freq,<EOL>sampling=sampling)<EOL>self.sampling = sampling<EOL>self._norm_aryule = norm<EOL>

**Constructor**

        For a detailled description of the parameters, see :func:`aryule`.

        :param array data: input data (list or numpy.array)
        :param int order:
        :param int NFFT: total length of the final data sets (padded with
            zero if needed; default is 4096)
        :param float sampling: sampling frequency of the input :attr:`data`
        :param str norm: don't change if you do not know


def _crosscov(x, y, axis=-<NUM_LIT:1>, all_lags=False, debias=True):

if x.shape[axis] != y.shape[axis]:<EOL><INDENT>raise ValueError(<EOL>'<STR_LIT>'<EOL>)<EOL><DEDENT>if debias:<EOL><INDENT>x = _remove_bias(x, axis)<EOL>y = _remove_bias(y, axis)<EOL><DEDENT>slicing = [slice(d) for d in x.shape]<EOL>slicing[axis] = slice(None,None,-<NUM_LIT:1>)<EOL>sxy = _fftconvolve(x, y[tuple(slicing)], axis=axis, mode='<STR_LIT>')<EOL>N = x.shape[axis]<EOL>sxy /= N<EOL>if all_lags:<EOL><INDENT>return sxy<EOL><DEDENT>slicing[axis] = slice(N-<NUM_LIT:1>,<NUM_LIT:2>*N-<NUM_LIT:1>)<EOL>return sxy[tuple(slicing)]<EOL>

Returns the crosscovariance sequence between two ndarrays.
    This is performed by calling fftconvolve on x, y[::-1]

    Parameters

    x: ndarray
    y: ndarray
    axis: time axis

    all_lags: {True/False}
    whether to return all nonzero lags, or to clip the length of s_xy
    to be the length of x and y. If False, then the zero lag covariance
    is at index 0. Otherwise, it is found at (len(x) + len(y) - 1)/2

    debias: {True/False}
    Always removes an estimate of the mean along the axis, unless
    told not to.

    cross covariance is defined as
    sxy[k] := E{X[t]*Y[t+k]}, where X,Y are zero mean random processes


<EOL>from scipy import linalg as la<EOL>Kmax = int(Kmax)<EOL>W = float(NW)/N<EOL>ab = np.zeros((<NUM_LIT:2>,N), '<STR_LIT:d>')<EOL>nidx = np.arange(N)<EOL>ab[<NUM_LIT:0>,<NUM_LIT:1>:] = nidx[<NUM_LIT:1>:]*(N-nidx[<NUM_LIT:1>:])/<NUM_LIT><EOL>ab[<NUM_LIT:1>] = ((N-<NUM_LIT:1>-<NUM_LIT:2>*nidx)/<NUM_LIT>)**<NUM_LIT:2> * np.cos(<NUM_LIT:2>*np.pi*W)<EOL>l,v = la.eig_banded(ab, select='<STR_LIT:i>', select_range=(N-Kmax, N-<NUM_LIT:1>))<EOL>dpss = v.transpose()[::-<NUM_LIT:1>]<EOL>fix_symmetric = (dpss[<NUM_LIT:0>::<NUM_LIT:2>].sum(axis=<NUM_LIT:1>) < <NUM_LIT:0>)<EOL>for i, f in enumerate(fix_symmetric):<EOL><INDENT>if f:<EOL><INDENT>dpss[<NUM_LIT:2>*i] *= -<NUM_LIT:1><EOL><DEDENT><DEDENT>fix_skew = (dpss[<NUM_LIT:1>::<NUM_LIT:2>,<NUM_LIT:1>] < <NUM_LIT:0>)<EOL>for i, f in enumerate(fix_skew):<EOL><INDENT>if f:<EOL><INDENT>dpss[<NUM_LIT:2>*i+<NUM_LIT:1>] *= -<NUM_LIT:1><EOL><DEDENT><DEDENT>acvs = _autocov(dpss, debias=False) * N<EOL>r = <NUM_LIT:4>*W*np.sinc(<NUM_LIT:2>*W*nidx)<EOL>r[<NUM_LIT:0>] = <NUM_LIT:2>*W<EOL>eigvals = np.dot(acvs, r)<EOL>return dpss, eigvals<EOL>

Returns the Discrete Prolate Spheroidal Sequences of orders [0,Kmax-1]
    for a given frequency-spacing multiple NW and sequence length N.

    See dpss function that is the official version. This version is indepedant
    of the C code and relies on Scipy function. However, it is slower by a factor 3

    Tridiagonal form of DPSS calculation from:


def CORRELOGRAMPSD(X, Y=None, lag=-<NUM_LIT:1>, window='<STR_LIT>',<EOL>norm='<STR_LIT>', NFFT=<NUM_LIT>, window_params={},<EOL>correlation_method='<STR_LIT>'):

N = len(X)<EOL>assert lag<N, '<STR_LIT>'<EOL>assert correlation_method in ['<STR_LIT>', '<STR_LIT>']<EOL>if Y is None:<EOL><INDENT>Y = numpy.array(X)<EOL>crosscorrelation = False<EOL><DEDENT>else:<EOL><INDENT>crosscorrelation = True<EOL><DEDENT>if NFFT is None:<EOL><INDENT>NFFT = N<EOL><DEDENT>psd = numpy.zeros(NFFT, dtype=complex)<EOL>w = Window(<NUM_LIT>*lag+<NUM_LIT:1>, window, **window_params)<EOL>w = w.data[lag+<NUM_LIT:1>:]<EOL>if correlation_method == '<STR_LIT>':<EOL><INDENT>rxy = CORRELATION (X, Y, maxlags=lag, norm=norm)<EOL><DEDENT>elif correlation_method == '<STR_LIT>':<EOL><INDENT>rxy, _l = xcorr (X, Y, maxlags=lag, norm=norm)<EOL>rxy = rxy[lag:]<EOL><DEDENT>psd[<NUM_LIT:0>] = rxy[<NUM_LIT:0>]<EOL>psd[<NUM_LIT:1>:lag+<NUM_LIT:1>] = rxy[<NUM_LIT:1>:] * w<EOL>if crosscorrelation is True:<EOL><INDENT>if correlation_method == '<STR_LIT>':<EOL><INDENT>ryx = CORRELATION(Y, X, maxlags=lag, norm=norm)<EOL><DEDENT>elif correlation_method == '<STR_LIT>':<EOL><INDENT>ryx, _l = xcorr(Y, X, maxlags=lag, norm=norm)<EOL>ryx = ryx[lag:]<EOL><DEDENT>psd[-<NUM_LIT:1>:NFFT-lag-<NUM_LIT:1>:-<NUM_LIT:1>] = ryx[<NUM_LIT:1>:].conjugate() * w<EOL><DEDENT>else: <EOL><INDENT>psd[-<NUM_LIT:1>:NFFT-lag-<NUM_LIT:1>:-<NUM_LIT:1>] = rxy[<NUM_LIT:1>:].conjugate() * w<EOL><DEDENT>psd = numpy.real(fft(psd))<EOL>return psd<EOL>

PSD estimate using correlogram method.

    :param array X: complex or real data samples X(1) to X(N)
    :param array Y: complex data samples Y(1) to Y(N). If provided, computes
        the cross PSD, otherwise the PSD is returned
    :param int lag: highest lag index to compute. Must be less than N
    :param str window_name: see :mod:`window` for list of valid names
    :param str norm: one of the valid normalisation of :func:`xcorr` (biased,
        unbiased, coeff, None)
    :param int NFFT: total length of the final data sets (padded with zero
        if needed; default is 4096)
    :param str correlation_method: either `xcorr` or `CORRELATION`.
        CORRELATION should be removed in the future.

    :return:
        * Array of real (cross) power spectral density estimate values. This is
          a two sided array with negative values following the positive ones
          whatever is the input data (real or complex).

    .. rubric:: Description:

    The exact power spectral density is the Fourier transform of the
    autocorrelation sequence:

    .. math:: P_{xx}(f) = T \sum_{m=-\infty}^{\infty} r_{xx}[m] exp^{-j2\pi fmT}

    The correlogram method of PSD estimation substitutes a finite sequence of
    autocorrelation estimates :math:`\hat{r}_{xx}` in place of :math:`r_{xx}`.
    This estimation can be computed with :func:`xcorr` or :func:`CORRELATION` by
    chosing a proprer lag `L`. The estimated PSD is then

    .. math:: \hat{P}_{xx}(f) = T \sum_{m=-L}^{L} \hat{r}_{xx}[m] exp^{-j2\pi fmT}

    The lag index must be less than the number of data samples `N`. Ideally, it
    should be around `L/10` [Marple]_ so as to avoid greater statistical
    variance associated with higher lags.

    To reduce the leakage of the implicit rectangular window and therefore to
    reduce the bias in the estimate, a tapering window is normally used and lead
    to the so-called Blackman and Tukey correlogram:

    .. math:: \hat{P}_{BT}(f) = T \sum_{m=-L}^{L} w[m] \hat{r}_{xx}[m] exp^{-j2\pi fmT}

    The correlogram for the cross power spectral estimate is

    .. math:: \hat{P}_{xx}(f) = T \sum_{m=-L}^{L} \hat{r}_{xx}[m] exp^{-j2\pi fmT}

    which is computed if :attr:`Y` is not provide. In such case,
    :math:`r_{yx} = r_{xy}` so we compute the correlation only once.

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import CORRELOGRAMPSD, marple_data
        from spectrum.tools import cshift
        from pylab import log10, axis, grid, plot,linspace

        psd = CORRELOGRAMPSD(marple_data, marple_data, lag=15)
        f = linspace(-0.5, 0.5, len(psd))
        psd = cshift(psd, len(psd)/2)
        plot(f, 10*log10(psd/max(psd)))
        axis([-0.5,0.5,-50,0])
        grid(True)

    .. seealso:: :func:`create_window`, :func:`CORRELATION`, :func:`xcorr`,
        :class:`pcorrelogram`.


token = os.urandom(<NUM_LIT:16>)<EOL>token_time = int(time.time())<EOL>return {'<STR_LIT>': token, '<STR_LIT>': token_time}<EOL>


try:<EOL><INDENT>return cls.get(cls.token == token)<EOL><DEDENT>except DoesNotExist:<EOL><INDENT>return None<EOL><DEDENT>


async def load_fk(self, info: SQLQueryInfo, records: Iterable[DataRecord]) -> Union[List, Iterable]:

<EOL>async def check(data, records):<EOL><INDENT>for column, fkvalues_lst in data.items():<EOL><INDENT>for fkvalues in fkvalues_lst:<EOL><INDENT>pks = []<EOL>all_ni = True<EOL>vcls = self.app.tables[fkvalues['<STR_LIT>']]<EOL>for i in records:<EOL><INDENT>val = i.get(column, NotImplemented)<EOL>if val != NotImplemented:<EOL><INDENT>all_ni = False<EOL><DEDENT>pks.append(val)<EOL><DEDENT>if all_ni:<EOL><INDENT>logger.debug("<STR_LIT>" % column)<EOL>continue<EOL><DEDENT>v = vcls(self.app, self._request)  <EOL>await v._prepare()<EOL>info2 = SQLQueryInfo()<EOL>info2.set_select(ALL_COLUMNS)<EOL>info2.add_condition(PRIMARY_KEY, SQL_OP.IN, pks)<EOL>info2.bind(v)<EOL>try:<EOL><INDENT>fk_records, count = await v._sql.select_page(info2, size=-<NUM_LIT:1>)<EOL><DEDENT>except RecordNotFound:<EOL><INDENT>continue<EOL><DEDENT>await v.check_records_permission(info2, fk_records)<EOL>fk_dict = {}<EOL>for i in fk_records:<EOL><INDENT>fk_dict[i[vcls.primary_key]] = i<EOL><DEDENT>column_to_set = fkvalues.get('<STR_LIT>', column) or column<EOL>for _, record in enumerate(records):<EOL><INDENT>k = record.get(column, NotImplemented)<EOL>if k in fk_dict:<EOL><INDENT>record[column_to_set] = fk_dict[k]<EOL><DEDENT><DEDENT>if fkvalues['<STR_LIT>']:<EOL><INDENT>await check(fkvalues['<STR_LIT>'], fk_records)<EOL><DEDENT><DEDENT><DEDENT><DEDENT>await check(info.loadfk, records)<EOL>return records<EOL>

:param info:
:param records: the data got from database and filtered from permission
:return:


async def after_delete(self, deleted_records: List[DataRecord]):

pass<EOL>


async def _call_handle(self, func, *args):

await async_call(func, *args)<EOL>if self.is_finished:<EOL><INDENT>raise FinishQuitException()<EOL><DEDENT>

call and check result of handle_query/read/insert/update


async def after_insert(self, raw_post: Dict, values: SQLValuesToWrite, record: DataRecord):

pass<EOL>

一对一
Emitted before finish
:param raw_post:
:param values:
:param record:
:return:


async def before_delete(self, records: List[DataRecord]):

pass<EOL>


pass<EOL>

All modules loaded, and ready to serve.
Emitted after register routes and before loop start
:return:


async def before_update(self, raw_post: Dict, values: SQLValuesToWrite, records: List[DataRecord]):

pass<EOL>

一对多，当有一个权限检查失败时即返回异常
raw_post 权限过滤和列过滤前，values 过滤后
:param raw_post:
:param values:
:param records:
:return:


@abstractmethod<EOL><INDENT>def setup_user_key(self, key, expires=<NUM_LIT:30>):<DEDENT>

pass<EOL>


pass<EOL>

teardown user key for server, make the token invalid here


pass<EOL>

Override to determine the current user from, e.g., a cookie.


def add_condition(self, field_name, op, value):

if not isinstance(op, SQL_OP):<EOL><INDENT>if op not in SQL_OP.txt2op:<EOL><INDENT>raise SQLOperatorInvalid(op)<EOL><DEDENT>else:<EOL><INDENT>op = SQL_OP.txt2op.get(op)<EOL><DEDENT><DEDENT>self.conditions.append([field_name, op, value])<EOL>

Add a query condition and validate it.
raise ParamsException if failed.
self.view required
:param field_name:
:param op:
:param value:
:return: None


orders = []<EOL>for i in map(str.strip, text.split('<STR_LIT:U+002C>')):<EOL><INDENT>items = i.split('<STR_LIT:.>', <NUM_LIT:2>)<EOL>if len(items) == <NUM_LIT:1>: column, order = items[<NUM_LIT:0>], '<STR_LIT:default>'<EOL>elif len(items) == <NUM_LIT:2>: column, order = items<EOL>else: raise InvalidParams("<STR_LIT>")<EOL>order = order.lower()<EOL>if order not in ('<STR_LIT>', '<STR_LIT>', '<STR_LIT:default>'):<EOL><INDENT>raise InvalidParams('<STR_LIT>' % order)<EOL><DEDENT>if order != '<STR_LIT:default>':<EOL><INDENT>orders.append(SQLQueryOrder(column, order))<EOL><DEDENT><DEDENT>return orders<EOL>

:param text: order=id.desc, xxx.asc
:return: [
    [<column>, asc|desc|default],
    [<column2>, asc|desc|default],
]


@classmethod<EOL><INDENT>def parse_load_fk(cls, data: Dict[str, List[Dict[str, object]]]) -> Dict[str, List[Dict[str, object]]]:<DEDENT>

default_value_dict = {'<STR_LIT>': None, '<STR_LIT>': None, '<STR_LIT>': None, '<STR_LIT>': None}<EOL>def value_normalize_dict(value):<EOL><INDENT>def check(k, v):<EOL><INDENT>if k == '<STR_LIT>': return isinstance(v, str)<EOL>if k == '<STR_LIT>': return isinstance(v, str)<EOL>if k == '<STR_LIT>': return isinstance(v, str)<EOL>if k == '<STR_LIT>': return isinstance(v, dict)<EOL><DEDENT>valid = {k: v for k, v in value.items() if check(k, v)}<EOL>if not valid: return default_value_dict.copy()<EOL>if '<STR_LIT>' in valid and valid['<STR_LIT>']:<EOL><INDENT>valid['<STR_LIT>'] = cls.parse_load_fk(valid['<STR_LIT>'])<EOL><DEDENT>for k, v in default_value_dict.items():<EOL><INDENT>valid.setdefault(k, v)<EOL><DEDENT>return valid<EOL><DEDENT>def value_normalize(value, no_list=True):<EOL><INDENT>if value is None:<EOL><INDENT>return default_value_dict.copy()<EOL><DEDENT>elif not no_list and isinstance(value, List):<EOL><INDENT>return list(map(value_normalize, value))<EOL><DEDENT>elif isinstance(value, str):<EOL><INDENT>val = default_value_dict.copy()<EOL>val['<STR_LIT>'] = value<EOL>return val<EOL><DEDENT>elif isinstance(value, Dict):<EOL><INDENT>return value_normalize_dict(value)<EOL><DEDENT>else:<EOL><INDENT>raise InvalidParams('<STR_LIT>' % value)<EOL><DEDENT><DEDENT>new_data = {}<EOL>if not isinstance(data, dict):<EOL><INDENT>raise InvalidParams('<STR_LIT>' % data)<EOL><DEDENT>for k, v in data.items():<EOL><INDENT>nv = value_normalize(v, False)<EOL>new_data[k] = nv if isinstance(nv, List) else [nv]<EOL><DEDENT>return new_data<EOL>

:param data:{
    <column>: role,
    <column2>: role,
    <column>: {
        'role': role,
        'loadfk': { ... },
    },
:return: {
    <column>: {
        'role': role,
    },
    ...
    <column3>: {
        'role': role,
        'loadfk': { ... },
    },
}


@classmethod<EOL><INDENT>def parse_select(cls, text: str) -> Set:<DEDENT>

if text == '<STR_LIT:*>':<EOL><INDENT>return ALL_COLUMNS  <EOL><DEDENT>selected_columns = set(filter(lambda x: x, map(str.strip, text.split('<STR_LIT:U+002C>'))))<EOL>if not selected_columns:<EOL><INDENT>raise InvalidParams("<STR_LIT>")<EOL><DEDENT>return selected_columns<EOL>

get columns from select text
:param text: col1, col2
:return: ALL_COLUMNS or ['col1', 'col2']


if isinstance(obj, str):<EOL><INDENT>if obj == '<STR_LIT:*>':<EOL><INDENT>return A.ALL<EOL><DEDENT>elif obj in A.ALL:<EOL><INDENT>return obj,<EOL><DEDENT>else:<EOL><INDENT>logger.warning('<STR_LIT>', obj)<EOL><DEDENT><DEDENT>elif isinstance(obj, (list, tuple)):<EOL><INDENT>for i in obj:<EOL><INDENT>if i not in A.ALL:<EOL><INDENT>logger.warning('<STR_LIT>', i)<EOL><DEDENT><DEDENT>return obj<EOL><DEDENT>elif isinstance(obj, dict):<EOL><INDENT>return self._parse_permission(obj.get('<STR_LIT:*>'))<EOL><DEDENT>

从 obj 中取出权限
:param obj:
:return: [A.QUERY, A.WRITE, ...]


def can_with_record(self, user, action, record: DataRecord, *, available=None):

assert action not in (A.QUERY, A.CREATE), "<STR_LIT>" % action<EOL>rules = []<EOL>for rule in self.record_checks:<EOL><INDENT>if record.table == rule[<NUM_LIT:0>] and action in rule[<NUM_LIT:1>]:<EOL><INDENT>rules.append(rule)<EOL><DEDENT><DEDENT>if available is None: available = self.can_with_columns(user, action, record.table, record.keys())<EOL>else: available = list(available)<EOL>bak = available.copy()<EOL>for rule in rules:<EOL><INDENT>ret = rule[-<NUM_LIT:1>](self, user, action, record, available)<EOL>if isinstance(ret, (tuple, set, list)):<EOL><INDENT>available = list(ret)<EOL><DEDENT>elif ret == '<STR_LIT:*>':<EOL><INDENT>available = list(bak)<EOL><DEDENT>elif not ret:<EOL><INDENT>available = []<EOL><DEDENT><DEDENT>return available<EOL>

进行基于 Record 的权限判定，返回可用列。
:param user:
:param action:
:param record:
:param available: 限定检查范围
:return: 可用列


def can_with_columns(self, user, action, table, columns):

<EOL>global_data = self.rules.get('<STR_LIT:*>')<EOL>global_actions = self._parse_permission(global_data)<EOL>if global_actions and action in global_actions:<EOL><INDENT>available = list(columns)<EOL><DEDENT>else:<EOL><INDENT>available = []<EOL><DEDENT>table_data = self.rules.get(table)<EOL>table_actions = self._parse_permission(table_data)<EOL>if table_actions and action in table_actions:<EOL><INDENT>available = list(columns)<EOL><DEDENT>if type(table_data) == dict:<EOL><INDENT>for column in columns:<EOL><INDENT>column_actions = self._parse_permission(table_data.get(column))<EOL>if column_actions is not None:<EOL><INDENT>if action in column_actions:<EOL><INDENT>if column not in available:<EOL><INDENT>available.append(column)<EOL><DEDENT><DEDENT>else:<EOL><INDENT>if column in available:<EOL><INDENT>available.remove(column)<EOL><DEDENT><DEDENT><DEDENT><DEDENT><DEDENT>for check in self.common_checks:<EOL><INDENT>if check[<NUM_LIT:0>] == table and action in check[<NUM_LIT:1>]:<EOL><INDENT>ret = check[-<NUM_LIT:1>](self, user, action, available)<EOL>if isinstance(ret, (tuple, set, list)):<EOL><INDENT>available = list(ret)<EOL><DEDENT>elif ret == '<STR_LIT:*>':<EOL><INDENT>available = list(columns)<EOL><DEDENT>elif ret is False:<EOL><INDENT>available = []<EOL><DEDENT>if not available: break<EOL><DEDENT><DEDENT>return available<EOL>

根据权限进行列过滤
注意一点，只要有一个条件能够通过权限检测，那么过滤后还会有剩余条件，最终就不会报错。
如果全部条件都不能过检测，就会爆出权限错误了。

:param user:
:param action: 行为
:param table: 表名
:param columns: 列名列表
:return: 可用列的列表


def add_common_check(self, actions, table, func):

self.common_checks.append([table, actions, func])<EOL>"""<STR_LIT>"""<EOL>

emitted before query
:param actions:
:param table:
:param func:
:return:


def add_static(self, prefix, path, **kwargs):

self.statics.append((prefix, path, kwargs),)<EOL>

:param prefix: URL prefix
:param path: file directory
:param kwargs:
:return:


def view_bind(app, cls_url, view_cls: Type['<STR_LIT>']):

if view_cls._no_route: return<EOL>cls_url = cls_url or view_cls.__class__.__name__.lower()<EOL>def add_route(name, route_info, beacon_info):<EOL><INDENT>for method in route_info['<STR_LIT>']:<EOL><INDENT>async def beacon(request): pass<EOL>route_key = route_info['<STR_LIT:url>'] if route_info['<STR_LIT:url>'] else name<EOL>app._raw_app.router.add_route(method, urljoin('<STR_LIT>', cls_url, route_key), beacon)<EOL>app.route._beacons[beacon] = beacon_info<EOL><DEDENT><DEDENT>for name, route_info_lst in view_cls._interface.items():<EOL><INDENT>for route_info in route_info_lst:<EOL><INDENT>real_handler = getattr(view_cls, name, None)<EOL>if real_handler is None: continue <EOL>assert real_handler is not None, "<STR_LIT>"<EOL>handler_name = '<STR_LIT>' % (view_cls.__name__, real_handler.__name__)<EOL>assert iscoroutinefunction(real_handler), "<STR_LIT>" % handler_name<EOL>beacon_info = {<EOL>'<STR_LIT>': view_cls,<EOL>'<STR_LIT:name>': name,<EOL>'<STR_LIT>': real_handler,<EOL>'<STR_LIT>': route_info<EOL>}<EOL>add_route(name, route_info, beacon_info)<EOL><DEDENT><DEDENT>

将 API 绑定到 web 服务上
:param view_cls:
:param app:
:param cls_url:
:return:


@classmethod<EOL><INDENT>async def get_session(cls, view):<DEDENT>

session = cls(view)<EOL>session.key = await session.get_key()<EOL>session._data = await session.load() or {}<EOL>return session<EOL>

Every request have a session instance
:param view:
:return:


fp = io.BytesIO()<EOL>_pack3(obj, fp, **options)<EOL>return fp.getvalue()<EOL>

Serialize a Python object into MessagePack bytes.

Args:
    obj: a Python object

Kwargs:
    ext_handlers (dict): dictionary of Ext handlers, mapping a custom type
                         to a callable that packs an instance of the type
                         into an Ext object
    force_float_precision (str): "single" to force packing floats as
                                 IEEE-754 single-precision floats,
                                 "double" to force packing floats as
                                 IEEE-754 double-precision floats.

Returns:
    A 'bytes' containing serialized MessagePack bytes.

Raises:
    UnsupportedType(PackException):
        Object type not supported for packing.

Example:
>>> umsgpack.packb({u"compact": True, u"schema": 0})
b'\x82\xa7compact\xc3\xa6schema\x00'
>>>


s = "<STR_LIT>" % self.type<EOL>s += "<STR_LIT:U+0020>".join(["<STR_LIT>" % ord(self.data[i:i + <NUM_LIT:1>])<EOL>for i in xrange(min(len(self.data), <NUM_LIT:8>))])<EOL>if len(self.data) > <NUM_LIT:8>:<EOL><INDENT>s += "<STR_LIT>"<EOL><DEDENT>s += "<STR_LIT:)>"<EOL>return s<EOL>

String representation of this Ext object.


return not self.__eq__(other)<EOL>

Compare this Ext object with another for inequality.


if not isinstance(s, (str, bytearray)):<EOL><INDENT>raise TypeError("<STR_LIT>")<EOL><DEDENT>return _unpack(io.BytesIO(s), options)<EOL>

Deserialize MessagePack bytes into a Python object.

Args:
    s: a 'str' or 'bytearray' containing serialized MessagePack bytes

Kwargs:
    ext_handlers (dict): dictionary of Ext handlers, mapping integer Ext
                         type to a callable that unpacks an instance of
                         Ext into an object
    use_ordered_dict (bool): unpack maps into OrderedDict, instead of
                             unordered dict (default False)
    allow_invalid_utf8 (bool): unpack invalid strings into instances of
                               InvalidString, for access to the bytes
                               (default False)

Returns:
    A Python object.

Raises:
    TypeError:
        Packed data type is neither 'str' nor 'bytearray'.
    InsufficientDataException(UnpackException):
        Insufficient data to unpack the serialized object.
    InvalidStringException(UnpackException):
        Invalid UTF-8 string encountered during unpacking.
    UnsupportedTimestampException(UnpackException):
        Unsupported timestamp format encountered during unpacking.
    ReservedCodeException(UnpackException):
        Reserved code encountered during unpacking.
    UnhashableKeyException(UnpackException):
        Unhashable key encountered during map unpacking.
        The serialized map cannot be deserialized into a Python dictionary.
    DuplicateKeyException(UnpackException):
        Duplicate key encountered during map unpacking.

Example:
>>> umsgpack.unpackb(b'\x82\xa7compact\xc3\xa6schema\x00')
{u'compact': True, u'schema': 0}
>>>


return hash((self.type, self.data))<EOL>


global compatibility<EOL>ext_handlers = options.get("<STR_LIT>")<EOL>if obj is None:<EOL><INDENT>_pack_nil(obj, fp, options)<EOL><DEDENT>elif ext_handlers and obj.__class__ in ext_handlers:<EOL><INDENT>_pack_ext(ext_handlers[obj.__class__](obj), fp, options)<EOL><DEDENT>elif isinstance(obj, bool):<EOL><INDENT>_pack_boolean(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, int):<EOL><INDENT>_pack_integer(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, float):<EOL><INDENT>_pack_float(obj, fp, options)<EOL><DEDENT>elif compatibility and isinstance(obj, str):<EOL><INDENT>_pack_oldspec_raw(obj.encode('<STR_LIT:utf-8>'), fp, options)<EOL><DEDENT>elif compatibility and isinstance(obj, bytes):<EOL><INDENT>_pack_oldspec_raw(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, str):<EOL><INDENT>_pack_string(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, bytes):<EOL><INDENT>_pack_binary(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, list) or isinstance(obj, tuple):<EOL><INDENT>_pack_array(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, dict):<EOL><INDENT>_pack_map(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, datetime.datetime):<EOL><INDENT>_pack_ext_timestamp(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, Ext):<EOL><INDENT>_pack_ext(obj, fp, options)<EOL><DEDENT>elif ext_handlers:<EOL><INDENT>t = next((t for t in ext_handlers.keys() if isinstance(obj, t)), None)<EOL>if t:<EOL><INDENT>_pack_ext(ext_handlers[t](obj), fp, options)<EOL><DEDENT>else:<EOL><INDENT>raise UnsupportedTypeException(<EOL>"<STR_LIT>" % str(type(obj)))<EOL><DEDENT><DEDENT>else:<EOL><INDENT>raise UnsupportedTypeException(<EOL>"<STR_LIT>" % str(type(obj)))<EOL><DEDENT>

Serialize a Python object into MessagePack bytes.

Args:
    obj: a Python object
    fp: a .write()-supporting file-like object

Kwargs:
    ext_handlers (dict): dictionary of Ext handlers, mapping a custom type
                         to a callable that packs an instance of the type
                         into an Ext object
    force_float_precision (str): "single" to force packing floats as
                                 IEEE-754 single-precision floats,
                                 "double" to force packing floats as
                                 IEEE-754 double-precision floats.

Returns:
    None.

Raises:
    UnsupportedType(PackException):
        Object type not supported for packing.

Example:
>>> f = open('test.bin', 'wb')
>>> umsgpack.pack({u"compact": True, u"schema": 0}, f)
>>>


fp = io.BytesIO()<EOL>_pack2(obj, fp, **options)<EOL>return fp.getvalue()<EOL>

Serialize a Python object into MessagePack bytes.

Args:
    obj: a Python object

Kwargs:
    ext_handlers (dict): dictionary of Ext handlers, mapping a custom type
                         to a callable that packs an instance of the type
                         into an Ext object
    force_float_precision (str): "single" to force packing floats as
                                 IEEE-754 single-precision floats,
                                 "double" to force packing floats as
                                 IEEE-754 double-precision floats.

Returns:
    A 'str' containing serialized MessagePack bytes.

Raises:
    UnsupportedType(PackException):
        Object type not supported for packing.

Example:
>>> umsgpack.packb({u"compact": True, u"schema": 0})
'\x82\xa7compact\xc3\xa6schema\x00'
>>>


global compatibility<EOL>ext_handlers = options.get("<STR_LIT>")<EOL>if obj is None:<EOL><INDENT>_pack_nil(obj, fp, options)<EOL><DEDENT>elif ext_handlers and obj.__class__ in ext_handlers:<EOL><INDENT>_pack_ext(ext_handlers[obj.__class__](obj), fp, options)<EOL><DEDENT>elif isinstance(obj, bool):<EOL><INDENT>_pack_boolean(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, int) or isinstance(obj, long):<EOL><INDENT>_pack_integer(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, float):<EOL><INDENT>_pack_float(obj, fp, options)<EOL><DEDENT>elif compatibility and isinstance(obj, unicode):<EOL><INDENT>_pack_oldspec_raw(bytes(obj), fp, options)<EOL><DEDENT>elif compatibility and isinstance(obj, bytes):<EOL><INDENT>_pack_oldspec_raw(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, unicode):<EOL><INDENT>_pack_string(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, str):<EOL><INDENT>_pack_binary(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, list) or isinstance(obj, tuple):<EOL><INDENT>_pack_array(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, dict):<EOL><INDENT>_pack_map(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, datetime.datetime):<EOL><INDENT>_pack_ext_timestamp(obj, fp, options)<EOL><DEDENT>elif isinstance(obj, Ext):<EOL><INDENT>_pack_ext(obj, fp, options)<EOL><DEDENT>elif ext_handlers:<EOL><INDENT>t = next((t for t in ext_handlers.keys() if isinstance(obj, t)), None)<EOL>if t:<EOL><INDENT>_pack_ext(ext_handlers[t](obj), fp, options)<EOL><DEDENT>else:<EOL><INDENT>raise UnsupportedTypeException(<EOL>"<STR_LIT>" % str(type(obj)))<EOL><DEDENT><DEDENT>else:<EOL><INDENT>raise UnsupportedTypeException("<STR_LIT>" % str(type(obj)))<EOL><DEDENT>


def pagination_calc(items_count, page_size, cur_page=<NUM_LIT:1>, nearby=<NUM_LIT:2>):

if type(cur_page) == str:<EOL><INDENT>cur_page = int(cur_page) if cur_page.isdigit() else <NUM_LIT:1><EOL><DEDENT>elif type(cur_page) == int:<EOL><INDENT>if cur_page <= <NUM_LIT:0>:<EOL><INDENT>cur_page = <NUM_LIT:1><EOL><DEDENT><DEDENT>else:<EOL><INDENT>cur_page = <NUM_LIT:1><EOL><DEDENT>page_count = <NUM_LIT:1> if page_size == -<NUM_LIT:1> else int(math.ceil(items_count / page_size))<EOL>items_length = nearby * <NUM_LIT:2> + <NUM_LIT:1><EOL>first_page = None<EOL>last_page = None<EOL>prev_page = cur_page - <NUM_LIT:1> if cur_page != <NUM_LIT:1> else None<EOL>next_page = cur_page + <NUM_LIT:1> if cur_page != page_count else None<EOL>if page_count <= items_length:<EOL><INDENT>items = range(<NUM_LIT:1>, page_count + <NUM_LIT:1>)<EOL><DEDENT>elif cur_page <= nearby:<EOL><INDENT>items = range(<NUM_LIT:1>, items_length + <NUM_LIT:1>)<EOL>last_page = True<EOL><DEDENT>elif cur_page >= page_count - nearby:<EOL><INDENT>items = range(page_count - items_length + <NUM_LIT:1>, page_count + <NUM_LIT:1>)<EOL>first_page = True<EOL><DEDENT>else:<EOL><INDENT>items = range(cur_page - nearby, cur_page + nearby + <NUM_LIT:1>)<EOL>first_page, last_page = True, True<EOL><DEDENT>if first_page:<EOL><INDENT>first_page = <NUM_LIT:1><EOL><DEDENT>if last_page:<EOL><INDENT>last_page = page_count<EOL><DEDENT>return {<EOL>'<STR_LIT>': cur_page,<EOL>'<STR_LIT>': prev_page,<EOL>'<STR_LIT>': next_page,<EOL>'<STR_LIT>': first_page,<EOL>'<STR_LIT>': last_page,<EOL>'<STR_LIT>': list(items),<EOL>'<STR_LIT:info>': {<EOL>'<STR_LIT>': page_size,<EOL>'<STR_LIT>': page_count,<EOL>'<STR_LIT>': items_count,<EOL>}<EOL>}<EOL>

:param nearby:
:param items_count: count of all items
:param page_size: size of one page
:param cur_page: current page number, accept string digit
:return: num of pages, an iterator


if isinstance(val, bytes):<EOL><INDENT>return val<EOL><DEDENT>elif isinstance(val, memoryview):<EOL><INDENT>return val.tobytes()<EOL><DEDENT>else:<EOL><INDENT>raise TypeError('<STR_LIT>')<EOL><DEDENT>

不同数据库从blob拿出的数据有所差别，有的是memoryview有的是bytes


data = json.loads(data)<EOL>for i in ('<STR_LIT>', '<STR_LIT>', '<STR_LIT>'):<EOL><INDENT>if i not in data:<EOL><INDENT>raise QueryException("<STR_LIT>" % i)<EOL><DEDENT><DEDENT>tables = data['<STR_LIT>']<EOL>columns = data['<STR_LIT>']<EOL>conditions = data['<STR_LIT>']<EOL>def parse_stmt(s, expr_cls, all_op, multi_items_op):<EOL><INDENT>if len(s) == <NUM_LIT:0>:<EOL><INDENT>return []<EOL><DEDENT>if s[<NUM_LIT:0>] in all_op:<EOL><INDENT>if s[<NUM_LIT:0>] in multi_items_op:<EOL><INDENT>values = []<EOL>for i in s[<NUM_LIT:1>:]:<EOL><INDENT>values.append(parse_stmt(i, expr_cls, all_op, multi_items_op))<EOL><DEDENT>return expr_cls(None, s[<NUM_LIT:0>], None, values=values)<EOL><DEDENT>else:<EOL><INDENT>if len(s) == <NUM_LIT:5>:<EOL><INDENT>lhs = Column(s[<NUM_LIT:2>], table=s[<NUM_LIT:1>])<EOL>rhs = Column(s[<NUM_LIT:4>], table=s[<NUM_LIT:3>])<EOL>if (s[<NUM_LIT:1>] not in tables) or (s[<NUM_LIT:3>] not in tables):<EOL><INDENT>raise QueryException('<STR_LIT>')<EOL><DEDENT>return expr_cls(lhs, s[<NUM_LIT:0>], rhs)<EOL><DEDENT>else:<EOL><INDENT>lhs = Column(s[<NUM_LIT:2>], table=s[<NUM_LIT:1>])<EOL>if s[<NUM_LIT:1>] not in tables:<EOL><INDENT>raise QueryException('<STR_LIT>')<EOL><DEDENT>return expr_cls(lhs, s[<NUM_LIT:0>], s[<NUM_LIT:3>])<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>raise QueryException('<STR_LIT>')<EOL><DEDENT><DEDENT>query_op = ('<STR_LIT:+>', '<STR_LIT:->', '<STR_LIT:*>', '<STR_LIT:/>')<EOL>query_columns = []<EOL>for i in columns:<EOL><INDENT>if len(i) == <NUM_LIT:2>:<EOL><INDENT>query_columns.append(Column(i[<NUM_LIT:1>], table=i[<NUM_LIT:0>]))<EOL><DEDENT>else:<EOL><INDENT>query_columns.append(parse_stmt(i, QueryExpression, query_op, query_op))<EOL><DEDENT><DEDENT>wheres = parse_stmt(conditions, ConditionExpression, _operator_map, ('<STR_LIT>', '<STR_LIT>',))<EOL>return {<EOL>'<STR_LIT>': tables,<EOL>'<STR_LIT>': query_columns,<EOL>'<STR_LIT>': wheres,<EOL>}<EOL>


if branch == MASTER_BRANCH:<EOL><INDENT>return<EOL><DEDENT>output = subprocess.check_output(['<STR_LIT>', '<STR_LIT>', '<STR_LIT>'])<EOL>for line in output.split('<STR_LIT:\n>'):<EOL><INDENT>if line.strip() == branch:<EOL><INDENT>break<EOL><DEDENT><DEDENT>else:<EOL><INDENT>try:<EOL><INDENT>sys.stdout.write(subprocess.check_output(<EOL>['<STR_LIT>', '<STR_LIT>', '<STR_LIT>', '<STR_LIT>' % branch]))<EOL><DEDENT>except subprocess.CalledProcessError:<EOL><INDENT>raise SystemExit(<NUM_LIT:1>)<EOL><DEDENT><DEDENT>

Track the specified remote branch if it is not already tracked.


def block_reduction_b(inputs, scope=None, is_train=False):

<EOL>with tf.variable_scope(scope, '<STR_LIT>', [inputs]):<EOL><INDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_0, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_0, _ = conv_module(<EOL>branch_0, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:2>, <NUM_LIT:2>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_1, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_1, _ = conv_module(<EOL>branch_1, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:7>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_1, _ = conv_module(<EOL>branch_1, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:7>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_1, _ = conv_module(<EOL>branch_1, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:2>, <NUM_LIT:2>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_2 = tl.layers.MaxPool2d(inputs, (<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:2>, <NUM_LIT:2>), padding='<STR_LIT>', name='<STR_LIT>')<EOL><DEDENT>return tl.layers.ConcatLayer([branch_0, branch_1, branch_2], concat_dim=<NUM_LIT:3>, name='<STR_LIT>')<EOL><DEDENT>

Builds Reduction-B block for Inception v4 network.


def block_inception_a(inputs, scope=None, is_train=False):

<EOL>with tf.variable_scope(name_or_scope=scope, default_name='<STR_LIT>', values=[inputs]):<EOL><INDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_0, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_1, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT:64>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_1, _ = conv_module(<EOL>branch_1, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_2, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT:64>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_2, _ = conv_module(<EOL>branch_2, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_2, _ = conv_module(<EOL>branch_2, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_3 = tl.layers.MeanPool2d(<EOL>inputs, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_3, _ = conv_module(<EOL>branch_3, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>return tl.layers.ConcatLayer([branch_0, branch_1, branch_2, branch_3], concat_dim=<NUM_LIT:3>, name='<STR_LIT>')<EOL><DEDENT>

Builds Inception-A block for Inception v4 network.


def block_inception_b(inputs, scope=None, is_train=False):

<EOL>with tf.variable_scope(scope, '<STR_LIT>', [inputs]):<EOL><INDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_0, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_1, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_1, _ = conv_module(<EOL>branch_1, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:7>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_1, _ = conv_module(<EOL>branch_1, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:7>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_2, _ = conv_module(<EOL>inputs, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_2, _ = conv_module(<EOL>branch_2, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:7>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_2, _ = conv_module(<EOL>branch_2, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:7>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_2, _ = conv_module(<EOL>branch_2, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:7>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_2, _ = conv_module(<EOL>branch_2, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:7>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>branch_3 = tl.layers.MeanPool2d(<EOL>inputs, filter_size=(<NUM_LIT:3>, <NUM_LIT:3>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL>branch_3, _ = conv_module(<EOL>branch_3, n_out_channel=<NUM_LIT>, filter_size=(<NUM_LIT:1>, <NUM_LIT:1>), strides=(<NUM_LIT:1>, <NUM_LIT:1>), padding='<STR_LIT>', batch_norm_init=None,<EOL>is_train=is_train, use_batchnorm=True, activation_fn='<STR_LIT>', name='<STR_LIT>'<EOL>)<EOL><DEDENT>return tl.layers.ConcatLayer([branch_0, branch_1, branch_2, branch_3], concat_dim=<NUM_LIT:3>, name='<STR_LIT>')<EOL><DEDENT>

Builds Inception-B block for Inception v4 network.


I = I[<NUM_LIT>:<NUM_LIT>]<EOL>I = I[::<NUM_LIT:2>, ::<NUM_LIT:2>, <NUM_LIT:0>]<EOL>I[I == <NUM_LIT>] = <NUM_LIT:0><EOL>I[I == <NUM_LIT>] = <NUM_LIT:0><EOL>I[I != <NUM_LIT:0>] = <NUM_LIT:1><EOL>return I.astype(np.float).ravel()<EOL>

Prepro 210x160x3 uint8 frame into 6400 (80x80) 1D float vector.


if FLAGS.model == "<STR_LIT>":<EOL><INDENT>init_scale = <NUM_LIT:0.1><EOL>learning_rate = <NUM_LIT:1.><EOL>max_grad_norm = <NUM_LIT:5><EOL>num_steps = <NUM_LIT:20><EOL>hidden_size = <NUM_LIT:200><EOL>max_epoch = <NUM_LIT:4><EOL>max_max_epoch = <NUM_LIT><EOL>keep_prob = <NUM_LIT:1.0><EOL>lr_decay = <NUM_LIT:0.5><EOL>batch_size = <NUM_LIT:20><EOL>vocab_size = <NUM_LIT><EOL><DEDENT>elif FLAGS.model == "<STR_LIT>":<EOL><INDENT>init_scale = <NUM_LIT><EOL>learning_rate = <NUM_LIT:1.0><EOL>max_grad_norm = <NUM_LIT:5><EOL>num_steps = <NUM_LIT><EOL>hidden_size = <NUM_LIT><EOL>max_epoch = <NUM_LIT:6><EOL>max_max_epoch = <NUM_LIT><EOL>keep_prob = <NUM_LIT:0.5><EOL>lr_decay = <NUM_LIT><EOL>batch_size = <NUM_LIT:20><EOL>vocab_size = <NUM_LIT><EOL><DEDENT>elif FLAGS.model == "<STR_LIT>":<EOL><INDENT>init_scale = <NUM_LIT><EOL>learning_rate = <NUM_LIT:1.0><EOL>max_grad_norm = <NUM_LIT:10><EOL>num_steps = <NUM_LIT><EOL>hidden_size = <NUM_LIT><EOL>max_epoch = <NUM_LIT><EOL>max_max_epoch = <NUM_LIT><EOL>keep_prob = <NUM_LIT><EOL>lr_decay = <NUM_LIT:1> / <NUM_LIT><EOL>batch_size = <NUM_LIT:20><EOL>vocab_size = <NUM_LIT><EOL><DEDENT>else:<EOL><INDENT>raise ValueError("<STR_LIT>", FLAGS.model)<EOL><DEDENT>train_data, valid_data, test_data, vocab_size = tl.files.load_ptb_dataset()<EOL>print('<STR_LIT>'.format(len(train_data)))  <EOL>print('<STR_LIT>'.format(len(valid_data)))  <EOL>print('<STR_LIT>'.format(len(test_data)))  <EOL>print('<STR_LIT>'.format(vocab_size))  <EOL>sess = tf.InteractiveSession()<EOL>input_data = tf.placeholder(tf.int32, [batch_size, num_steps])<EOL>targets = tf.placeholder(tf.int32, [batch_size, num_steps])<EOL>input_data_test = tf.placeholder(tf.int32, [<NUM_LIT:1>, <NUM_LIT:1>])<EOL>targets_test = tf.placeholder(tf.int32, [<NUM_LIT:1>, <NUM_LIT:1>])<EOL>def inference(x, is_training, num_steps, reuse=None):<EOL><INDENT>"""<STR_LIT>"""<EOL>print("<STR_LIT>" % (num_steps, is_training, reuse))<EOL>init = tf.random_uniform_initializer(-init_scale, init_scale)<EOL>with tf.variable_scope("<STR_LIT>", reuse=reuse):<EOL><INDENT>net = tl.layers.EmbeddingInputlayer(x, vocab_size, hidden_size, init, name='<STR_LIT>')<EOL>net = tl.layers.DropoutLayer(net, keep=keep_prob, is_fix=True, is_train=is_training, name='<STR_LIT>')<EOL>net = tl.layers.RNNLayer(<EOL>net,<EOL>cell_fn=tf.contrib.rnn.BasicLSTMCell,  <EOL>cell_init_args={<EOL>'<STR_LIT>': <NUM_LIT:0.0>,<EOL>'<STR_LIT>': True<EOL>},<EOL>n_hidden=hidden_size,<EOL>initializer=init,<EOL>n_steps=num_steps,<EOL>return_last=False,<EOL>name='<STR_LIT>'<EOL>)<EOL>lstm1 = net<EOL>net = tl.layers.DropoutLayer(net, keep=keep_prob, is_fix=True, is_train=is_training, name='<STR_LIT>')<EOL>net = tl.layers.RNNLayer(<EOL>net,<EOL>cell_fn=tf.contrib.rnn.BasicLSTMCell,  <EOL>cell_init_args={<EOL>'<STR_LIT>': <NUM_LIT:0.0>,<EOL>'<STR_LIT>': True<EOL>},<EOL>n_hidden=hidden_size,<EOL>initializer=init,<EOL>n_steps=num_steps,<EOL>return_last=False,<EOL>return_seq_2d=True,<EOL>name='<STR_LIT>'<EOL>)<EOL>lstm2 = net<EOL>net = tl.layers.DropoutLayer(net, keep=keep_prob, is_fix=True, is_train=is_training, name='<STR_LIT>')<EOL>net = tl.layers.DenseLayer(net, vocab_size, W_init=init, b_init=init, act=None, name='<STR_LIT>')<EOL><DEDENT>return net, lstm1, lstm2<EOL><DEDENT>net, lstm1, lstm2 = inference(input_data, is_training=True, num_steps=num_steps, reuse=None)<EOL>net_val, lstm1_val, lstm2_val = inference(input_data, is_training=False, num_steps=num_steps, reuse=True)<EOL>net_test, lstm1_test, lstm2_test = inference(input_data_test, is_training=False, num_steps=<NUM_LIT:1>, reuse=True)<EOL>sess.run(tf.global_variables_initializer())<EOL>def loss_fn(outputs, targets, batch_size):<EOL><INDENT>loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example(<EOL>[outputs], [tf.reshape(targets, [-<NUM_LIT:1>])], [tf.ones_like(tf.reshape(targets, [-<NUM_LIT:1>]), dtype=tf.float32)]<EOL>)<EOL>cost = tf.reduce_sum(loss) / batch_size<EOL>return cost<EOL><DEDENT>cost = loss_fn(net.outputs, targets, batch_size)<EOL>cost_val = loss_fn(net_val.outputs, targets, batch_size)<EOL>cost_test = loss_fn(net_test.outputs, targets_test, <NUM_LIT:1>)<EOL>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>lr = tf.Variable(<NUM_LIT:0.0>, trainable=False)<EOL><DEDENT>tvars = tf.trainable_variables()<EOL>grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), max_grad_norm)<EOL>optimizer = tf.train.GradientDescentOptimizer(lr)<EOL>train_op = optimizer.apply_gradients(zip(grads, tvars))<EOL>sess.run(tf.global_variables_initializer())<EOL>net.print_params()<EOL>net.print_layers()<EOL>tl.layers.print_all_variables()<EOL>print("<STR_LIT>")<EOL>for i in range(max_max_epoch):<EOL><INDENT>new_lr_decay = lr_decay**max(i - max_epoch, <NUM_LIT:0.0>)<EOL>sess.run(tf.assign(lr, learning_rate * new_lr_decay))<EOL>print("<STR_LIT>" % (i + <NUM_LIT:1>, max_max_epoch, sess.run(lr)))<EOL>epoch_size = ((len(train_data) // batch_size) - <NUM_LIT:1>) // num_steps<EOL>start_time = time.time()<EOL>costs = <NUM_LIT:0.0><EOL>iters = <NUM_LIT:0><EOL>state1 = tl.layers.initialize_rnn_state(lstm1.initial_state)<EOL>state2 = tl.layers.initialize_rnn_state(lstm2.initial_state)<EOL>for step, (x, y) in enumerate(tl.iterate.ptb_iterator(train_data, batch_size, num_steps)):<EOL><INDENT>feed_dict = {<EOL>input_data: x,<EOL>targets: y,<EOL>lstm1.initial_state.c: state1[<NUM_LIT:0>],<EOL>lstm1.initial_state.h: state1[<NUM_LIT:1>],<EOL>lstm2.initial_state.c: state2[<NUM_LIT:0>],<EOL>lstm2.initial_state.h: state2[<NUM_LIT:1>],<EOL>}<EOL>feed_dict.update(net.all_drop)<EOL>_cost, state1_c, state1_h, state2_c, state2_h, _ = sess.run(<EOL>[cost, lstm1.final_state.c, lstm1.final_state.h, lstm2.final_state.c, lstm2.final_state.h, train_op],<EOL>feed_dict=feed_dict<EOL>)<EOL>state1 = (state1_c, state1_h)<EOL>state2 = (state2_c, state2_h)<EOL>costs += _cost<EOL>iters += num_steps<EOL>if step % (epoch_size // <NUM_LIT:10>) == <NUM_LIT:10>:<EOL><INDENT>print(<EOL>"<STR_LIT>" %<EOL>(step * <NUM_LIT:1.0> / epoch_size, np.exp(costs / iters), iters * batch_size / (time.time() - start_time))<EOL>)<EOL><DEDENT><DEDENT>train_perplexity = np.exp(costs / iters)<EOL>print("<STR_LIT>" % (i + <NUM_LIT:1>, max_max_epoch, train_perplexity))<EOL>start_time = time.time()<EOL>costs = <NUM_LIT:0.0><EOL>iters = <NUM_LIT:0><EOL>state1 = tl.layers.initialize_rnn_state(lstm1_val.initial_state)<EOL>state2 = tl.layers.initialize_rnn_state(lstm2_val.initial_state)<EOL>for step, (x, y) in enumerate(tl.iterate.ptb_iterator(valid_data, batch_size, num_steps)):<EOL><INDENT>feed_dict = {<EOL>input_data: x,<EOL>targets: y,<EOL>lstm1_val.initial_state.c: state1[<NUM_LIT:0>],<EOL>lstm1_val.initial_state.h: state1[<NUM_LIT:1>],<EOL>lstm2_val.initial_state.c: state2[<NUM_LIT:0>],<EOL>lstm2_val.initial_state.h: state2[<NUM_LIT:1>],<EOL>}<EOL>_cost, state1_c, state1_h, state2_c, state2_h, _ = sess.run(<EOL>[<EOL>cost_val, lstm1_val.final_state.c, lstm1_val.final_state.h, lstm2_val.final_state.c,<EOL>lstm2_val.final_state.h,<EOL>tf.no_op()<EOL>], feed_dict=feed_dict<EOL>)<EOL>state1 = (state1_c, state1_h)<EOL>state2 = (state2_c, state2_h)<EOL>costs += _cost<EOL>iters += num_steps<EOL><DEDENT>valid_perplexity = np.exp(costs / iters)<EOL>print("<STR_LIT>" % (i + <NUM_LIT:1>, max_max_epoch, valid_perplexity))<EOL><DEDENT>print("<STR_LIT>")<EOL>start_time = time.time()<EOL>costs = <NUM_LIT:0.0><EOL>iters = <NUM_LIT:0><EOL>state1 = tl.layers.initialize_rnn_state(lstm1_test.initial_state)<EOL>state2 = tl.layers.initialize_rnn_state(lstm2_test.initial_state)<EOL>for step, (x, y) in enumerate(tl.iterate.ptb_iterator(test_data, batch_size=<NUM_LIT:1>, num_steps=<NUM_LIT:1>)):<EOL><INDENT>feed_dict = {<EOL>input_data_test: x,<EOL>targets_test: y,<EOL>lstm1_test.initial_state.c: state1[<NUM_LIT:0>],<EOL>lstm1_test.initial_state.h: state1[<NUM_LIT:1>],<EOL>lstm2_test.initial_state.c: state2[<NUM_LIT:0>],<EOL>lstm2_test.initial_state.h: state2[<NUM_LIT:1>],<EOL>}<EOL>_cost, state1_c, state1_h, state2_c, state2_h = sess.run(<EOL>[<EOL>cost_test,<EOL>lstm1_test.final_state.c,<EOL>lstm1_test.final_state.h,<EOL>lstm2_test.final_state.c,<EOL>lstm2_test.final_state.h,<EOL>], feed_dict=feed_dict<EOL>)<EOL>state1 = (state1_c, state1_h)<EOL>state2 = (state2_c, state2_h)<EOL>costs += _cost<EOL>iters += <NUM_LIT:1><EOL><DEDENT>test_perplexity = np.exp(costs / iters)<EOL>print("<STR_LIT>" % (test_perplexity, time.time() - start_time))<EOL>print(<EOL>"<STR_LIT>"<EOL>)<EOL>

The core of the model consists of an LSTM cell that processes one word at
a time and computes probabilities of the possible continuations of the
sentence. The memory state of the network is initialized with a vector
of zeros and gets updated after reading each word. Also, for computational
reasons, we will process data in mini-batches of size batch_size.


<EOL>vocabulary_size = <NUM_LIT><EOL>embedding_size = <NUM_LIT><EOL>model_file_name = "<STR_LIT>"<EOL>batch_size = None<EOL>print("<STR_LIT>")<EOL>all_var = tl.files.load_npy_to_any(name=model_file_name + '<STR_LIT>')<EOL>data = all_var['<STR_LIT:data>']<EOL>count = all_var['<STR_LIT:count>']<EOL>dictionary = all_var['<STR_LIT>']<EOL>reverse_dictionary = all_var['<STR_LIT>']<EOL>tl.nlp.save_vocab(count, name='<STR_LIT>' + model_file_name + '<STR_LIT>')<EOL>del all_var, data, count<EOL>load_params = tl.files.load_npz(name=model_file_name + '<STR_LIT>')<EOL>x = tf.placeholder(tf.int32, shape=[batch_size])<EOL>emb_net = tl.layers.EmbeddingInputlayer(x, vocabulary_size, embedding_size, name='<STR_LIT>')<EOL>sess.run(tf.global_variables_initializer())<EOL>tl.files.assign_params(sess, [load_params[<NUM_LIT:0>]], emb_net)<EOL>emb_net.print_params()<EOL>emb_net.print_layers()<EOL>word = b'<STR_LIT:hello>'<EOL>word_id = dictionary[word]<EOL>print('<STR_LIT>', word_id)<EOL>words = [b'<STR_LIT:i>', b'<STR_LIT>', b'<STR_LIT>', b'<STR_LIT>']<EOL>word_ids = tl.nlp.words_to_word_ids(words, dictionary, _UNK)<EOL>context = tl.nlp.word_ids_to_words(word_ids, reverse_dictionary)<EOL>print('<STR_LIT>', word_ids)<EOL>print('<STR_LIT>', context)<EOL>vector = sess.run(emb_net.outputs, feed_dict={x: [word_id]})<EOL>print('<STR_LIT>', vector.shape)<EOL>vectors = sess.run(emb_net.outputs, feed_dict={x: word_ids})<EOL>print('<STR_LIT>', vectors.shape)<EOL>

How to use Embedding layer, and how to convert IDs to vector,
    IDs to words, etc.


<EOL>init_scale = <NUM_LIT:0.1><EOL>learning_rate = <NUM_LIT:1.0><EOL>max_grad_norm = <NUM_LIT:5><EOL>sequence_length = <NUM_LIT:20><EOL>hidden_size = <NUM_LIT:200><EOL>max_epoch = <NUM_LIT:4><EOL>max_max_epoch = <NUM_LIT:100><EOL>lr_decay = <NUM_LIT><EOL>batch_size = <NUM_LIT:20><EOL>top_k_list = [<NUM_LIT:1>, <NUM_LIT:3>, <NUM_LIT:5>, <NUM_LIT:10>]<EOL>print_length = <NUM_LIT:30><EOL>model_file_name = "<STR_LIT>"<EOL>words = customized_read_words(input_fpath="<STR_LIT>")<EOL>vocab = tl.nlp.create_vocab([words], word_counts_output_file='<STR_LIT>', min_word_count=<NUM_LIT:1>)<EOL>vocab = tl.nlp.Vocabulary('<STR_LIT>', unk_word="<STR_LIT>")<EOL>vocab_size = vocab.unk_id + <NUM_LIT:1><EOL>train_data = [vocab.word_to_id(word) for word in words]<EOL>seed = "<STR_LIT>"<EOL>seed = nltk.tokenize.word_tokenize(seed)<EOL>print('<STR_LIT>' % seed)<EOL>sess = tf.InteractiveSession()<EOL>input_data = tf.placeholder(tf.int32, [batch_size, sequence_length])<EOL>targets = tf.placeholder(tf.int32, [batch_size, sequence_length])<EOL>input_data_test = tf.placeholder(tf.int32, [<NUM_LIT:1>, <NUM_LIT:1>])<EOL>def inference(x, is_train, sequence_length, reuse=None):<EOL><INDENT>"""<STR_LIT>"""<EOL>print("<STR_LIT>" % (sequence_length, is_train, reuse))<EOL>rnn_init = tf.random_uniform_initializer(-init_scale, init_scale)<EOL>with tf.variable_scope("<STR_LIT>", reuse=reuse):<EOL><INDENT>network = EmbeddingInputlayer(x, vocab_size, hidden_size, rnn_init, name='<STR_LIT>')<EOL>network = RNNLayer(<EOL>network, cell_fn=tf.contrib.rnn.BasicLSTMCell, cell_init_args={<EOL>'<STR_LIT>': <NUM_LIT:0.0>,<EOL>'<STR_LIT>': True<EOL>}, n_hidden=hidden_size, initializer=rnn_init, n_steps=sequence_length, return_last=False,<EOL>return_seq_2d=True, name='<STR_LIT>'<EOL>)<EOL>lstm1 = network<EOL>network = DenseLayer(network, vocab_size, W_init=rnn_init, b_init=rnn_init, act=None, name='<STR_LIT>')<EOL><DEDENT>return network, lstm1<EOL><DEDENT>network, lstm1 = inference(input_data, is_train=True, sequence_length=sequence_length, reuse=None)<EOL>network_test, lstm1_test = inference(input_data_test, is_train=False, sequence_length=<NUM_LIT:1>, reuse=True)<EOL>y_linear = network_test.outputs<EOL>y_soft = tf.nn.softmax(y_linear)<EOL>def loss_fn(outputs, targets, batch_size, sequence_length):<EOL><INDENT>loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example(<EOL>[outputs], [tf.reshape(targets, [-<NUM_LIT:1>])], [tf.ones([batch_size * sequence_length])]<EOL>)<EOL>cost = tf.reduce_sum(loss) / batch_size<EOL>return cost<EOL><DEDENT>cost = loss_fn(network.outputs, targets, batch_size, sequence_length)<EOL>with tf.variable_scope('<STR_LIT>'):<EOL><INDENT>lr = tf.Variable(<NUM_LIT:0.0>, trainable=False)<EOL><DEDENT>tvars = network.all_params<EOL>grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), max_grad_norm)<EOL>optimizer = tf.train.GradientDescentOptimizer(lr)<EOL>train_op = optimizer.apply_gradients(zip(grads, tvars))<EOL>sess.run(tf.global_variables_initializer())<EOL>print("<STR_LIT>")<EOL>for i in range(max_max_epoch):<EOL><INDENT>new_lr_decay = lr_decay**max(i - max_epoch, <NUM_LIT:0.0>)<EOL>sess.run(tf.assign(lr, learning_rate * new_lr_decay))<EOL>print("<STR_LIT>" % (i + <NUM_LIT:1>, max_max_epoch, sess.run(lr)))<EOL>epoch_size = ((len(train_data) // batch_size) - <NUM_LIT:1>) // sequence_length<EOL>start_time = time.time()<EOL>costs = <NUM_LIT:0.0><EOL>iters = <NUM_LIT:0><EOL>state1 = tl.layers.initialize_rnn_state(lstm1.initial_state)<EOL>for step, (x, y) in enumerate(tl.iterate.ptb_iterator(train_data, batch_size, sequence_length)):<EOL><INDENT>_cost, state1, _ = sess.run(<EOL>[cost, lstm1.final_state, train_op], feed_dict={<EOL>input_data: x,<EOL>targets: y,<EOL>lstm1.initial_state: state1<EOL>}<EOL>)<EOL>costs += _cost<EOL>iters += sequence_length<EOL>if step % (epoch_size // <NUM_LIT:10>) == <NUM_LIT:1>:<EOL><INDENT>print(<EOL>"<STR_LIT>" %<EOL>(step * <NUM_LIT:1.0> / epoch_size, np.exp(costs / iters), iters * batch_size / (time.time() - start_time))<EOL>)<EOL><DEDENT><DEDENT>train_perplexity = np.exp(costs / iters)<EOL>print("<STR_LIT>" % (i + <NUM_LIT:1>, max_max_epoch, train_perplexity))<EOL>for top_k in top_k_list:<EOL><INDENT>state1 = tl.layers.initialize_rnn_state(lstm1_test.initial_state)<EOL>outs_id = [vocab.word_to_id(w) for w in seed]<EOL>for ids in outs_id[:-<NUM_LIT:1>]:<EOL><INDENT>a_id = np.asarray(ids).reshape(<NUM_LIT:1>, <NUM_LIT:1>)<EOL>state1 = sess.run(<EOL>[lstm1_test.final_state], feed_dict={<EOL>input_data_test: a_id,<EOL>lstm1_test.initial_state: state1<EOL>}<EOL>)<EOL><DEDENT>a_id = outs_id[-<NUM_LIT:1>]<EOL>for _ in range(print_length):<EOL><INDENT>a_id = np.asarray(a_id).reshape(<NUM_LIT:1>, <NUM_LIT:1>)<EOL>out, state1 = sess.run(<EOL>[y_soft, lstm1_test.final_state], feed_dict={<EOL>input_data_test: a_id,<EOL>lstm1_test.initial_state: state1<EOL>}<EOL>)<EOL>a_id = tl.nlp.sample_top(out[<NUM_LIT:0>], top_k=top_k)<EOL>outs_id.append(a_id)<EOL><DEDENT>sentence = [vocab.id_to_word(w) for w in outs_id]<EOL>sentence = "<STR_LIT:U+0020>".join(sentence)<EOL>print(top_k, '<STR_LIT::>', sentence)<EOL><DEDENT><DEDENT>print("<STR_LIT>")<EOL>tl.files.save_npz(network_test.all_params, name=model_file_name)<EOL>

Generate text by Synced sequence input and output.


def data_to_tfrecord(images, labels, filename):

if os.path.isfile(filename):<EOL><INDENT>print("<STR_LIT>" % filename)<EOL>return<EOL><DEDENT>print("<STR_LIT>" % filename)<EOL>writer = tf.python_io.TFRecordWriter(filename)<EOL>for index, img in enumerate(images):<EOL><INDENT>img_raw = img.tobytes()<EOL>label = int(labels[index])<EOL>example = tf.train.Example(<EOL>features=tf.train.Features(<EOL>feature={<EOL>"<STR_LIT:label>": tf.train.Feature(int64_list=tf.train.Int64List(value=[label])),<EOL>'<STR_LIT>': tf.train.Feature(bytes_list=tf.train.BytesList(value=[img_raw])),<EOL>}<EOL>)<EOL>)<EOL>writer.write(example.SerializeToString())  <EOL><DEDENT>writer.close()<EOL>


filename_queue = tf.train.string_input_producer([filename])<EOL>reader = tf.TFRecordReader()<EOL>_, serialized_example = reader.read(filename_queue)<EOL>features = tf.parse_single_example(<EOL>serialized_example, features={<EOL>'<STR_LIT:label>': tf.FixedLenFeature([], tf.int64),<EOL>'<STR_LIT>': tf.FixedLenFeature([], tf.string),<EOL>}<EOL>)<EOL>img = tf.decode_raw(features['<STR_LIT>'], tf.float32)<EOL>img = tf.reshape(img, [<NUM_LIT:32>, <NUM_LIT:32>, <NUM_LIT:3>])<EOL>if is_train ==True:<EOL><INDENT>img = tf.random_crop(img, [<NUM_LIT>, <NUM_LIT>, <NUM_LIT:3>])<EOL>img = tf.image.random_flip_left_right(img)<EOL>img = tf.image.random_brightness(img, max_delta=<NUM_LIT>)<EOL>img = tf.image.random_contrast(img, lower=<NUM_LIT>, upper=<NUM_LIT>)<EOL>img = tf.image.per_image_standardization(img)<EOL><DEDENT>elif is_train == False:<EOL><INDENT>img = tf.image.resize_image_with_crop_or_pad(img, <NUM_LIT>, <NUM_LIT>)<EOL>img = tf.image.per_image_standardization(img)<EOL><DEDENT>elif is_train == None:<EOL><INDENT>img = img<EOL><DEDENT>label = tf.cast(features['<STR_LIT:label>'], tf.int32)<EOL>return img, label<EOL>


<EOL>with tf.name_scope("<STR_LIT>"):  <EOL><INDENT>image = tf.image.random_flip_left_right(image)<EOL><DEDENT>color_ordering = thread_id % <NUM_LIT:2><EOL>with tf.name_scope("<STR_LIT>"):  <EOL><INDENT>if color_ordering == <NUM_LIT:0>:<EOL><INDENT>image = tf.image.random_brightness(image, max_delta=<NUM_LIT> / <NUM_LIT>)<EOL>image = tf.image.random_saturation(image, lower=<NUM_LIT:0.5>, upper=<NUM_LIT>)<EOL>image = tf.image.random_hue(image, max_delta=<NUM_LIT>)<EOL>image = tf.image.random_contrast(image, lower=<NUM_LIT:0.5>, upper=<NUM_LIT>)<EOL><DEDENT>elif color_ordering == <NUM_LIT:1>:<EOL><INDENT>image = tf.image.random_brightness(image, max_delta=<NUM_LIT> / <NUM_LIT>)<EOL>image = tf.image.random_contrast(image, lower=<NUM_LIT:0.5>, upper=<NUM_LIT>)<EOL>image = tf.image.random_saturation(image, lower=<NUM_LIT:0.5>, upper=<NUM_LIT>)<EOL>image = tf.image.random_hue(image, max_delta=<NUM_LIT>)<EOL><DEDENT>image = tf.clip_by_value(image, <NUM_LIT:0.0>, <NUM_LIT:1.0>)<EOL><DEDENT>return image<EOL>

Perform random distortions on an image.
    Args:
        image: A float32 Tensor of shape [height, width, 3] with values in [0, 1).
        thread_id: Preprocessing thread id used to select the ordering of color
        distortions. There should be a multiple of 2 preprocessing threads.
    Returns:````
        distorted_image: A float32 Tensor of shape [height, width, 3] with values in
        [0, 1].


<EOL>return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))<EOL>

Wrapper for inserting a bytes Feature into a SequenceExample proto,
    e.g, an image in byte


def prefetch_input_data(<EOL>reader, file_pattern, is_training, batch_size, values_per_shard, input_queue_capacity_factor=<NUM_LIT:16>,<EOL>num_reader_threads=<NUM_LIT:1>, shard_queue_name="<STR_LIT>", value_queue_name="<STR_LIT>"<EOL>):

data_files = []<EOL>for pattern in file_pattern.split("<STR_LIT:U+002C>"):<EOL><INDENT>data_files.extend(tf.gfile.Glob(pattern))<EOL><DEDENT>if not data_files:<EOL><INDENT>tl.logging.fatal("<STR_LIT>", file_pattern)<EOL><DEDENT>else:<EOL><INDENT>tl.logging.info("<STR_LIT>", len(data_files), file_pattern)<EOL><DEDENT>if is_training:<EOL><INDENT>print("<STR_LIT>")<EOL>filename_queue = tf.train.string_input_producer(data_files, shuffle=True, capacity=<NUM_LIT:16>, name=shard_queue_name)<EOL>min_queue_examples = values_per_shard * input_queue_capacity_factor<EOL>capacity = min_queue_examples + <NUM_LIT:100> * batch_size<EOL>values_queue = tf.RandomShuffleQueue(<EOL>capacity=capacity, min_after_dequeue=min_queue_examples, dtypes=[tf.string],<EOL>name="<STR_LIT>" + value_queue_name<EOL>)<EOL><DEDENT>else:<EOL><INDENT>print("<STR_LIT>")<EOL>filename_queue = tf.train.string_input_producer(data_files, shuffle=False, capacity=<NUM_LIT:1>, name=shard_queue_name)<EOL>capacity = values_per_shard + <NUM_LIT:3> * batch_size<EOL>values_queue = tf.FIFOQueue(capacity=capacity, dtypes=[tf.string], name="<STR_LIT>" + value_queue_name)<EOL><DEDENT>enqueue_ops = []<EOL>for _ in range(num_reader_threads):<EOL><INDENT>_, value = reader.read(filename_queue)<EOL>enqueue_ops.append(values_queue.enqueue([value]))<EOL><DEDENT>tf.train.queue_runner.add_queue_runner(tf.train.queue_runner.QueueRunner(values_queue, enqueue_ops))<EOL>tf.summary.scalar(<EOL>"<STR_LIT>" % (values_queue.name, capacity),<EOL>tf.cast(values_queue.size(), tf.float32) * (<NUM_LIT:1.> / capacity)<EOL>)<EOL>return values_queue<EOL>

Prefetches string values from disk into an input queue.

    In training the capacity of the queue is important because a larger queue
    means better mixing of training examples between shards. The minimum number of
    values kept in the queue is values_per_shard * input_queue_capacity_factor,
    where input_queue_memory factor should be chosen to trade-off better mixing
    with memory usage.

    Args:
        reader: Instance of tf.ReaderBase.
        file_pattern: Comma-separated list of file patterns (e.g.
            /tmp/train_data-?????-of-00100).
        is_training: Boolean; whether prefetching for training or eval.
        batch_size: Model batch size used to determine queue capacity.
        values_per_shard: Approximate number of values per shard.
        input_queue_capacity_factor: Minimum number of values to keep in the queue
        in multiples of values_per_shard. See comments above.
        num_reader_threads: Number of reader threads to fill the queue.
        shard_queue_name: Name for the shards filename queue.
        value_queue_name: Name for the values input queue.

    Returns:
        A Queue containing prefetched string values.


transform_matrix = create_transformation_matrix()<EOL>result = tl.prepro.affine_transform_cv2(image, transform_matrix)  <EOL>coords = [[(<NUM_LIT:50>, <NUM_LIT:100>), (<NUM_LIT:100>, <NUM_LIT:100>), (<NUM_LIT:100>, <NUM_LIT:50>), (<NUM_LIT:200>, <NUM_LIT:200>)], [(<NUM_LIT>, <NUM_LIT:50>), (<NUM_LIT:200>, <NUM_LIT:50>), (<NUM_LIT:200>, <NUM_LIT:100>)]]<EOL>coords_result = tl.prepro.affine_transform_keypoints(coords, transform_matrix)<EOL>def imwrite(image, coords_list, name):<EOL><INDENT>coords_list_ = []<EOL>for coords in coords_list:<EOL><INDENT>coords = np.array(coords, np.int32)<EOL>coords = coords.reshape((-<NUM_LIT:1>, <NUM_LIT:1>, <NUM_LIT:2>))<EOL>coords_list_.append(coords)<EOL><DEDENT>image = cv2.polylines(image, coords_list_, True, (<NUM_LIT:0>, <NUM_LIT:255>, <NUM_LIT:255>), <NUM_LIT:3>)<EOL>cv2.imwrite(name, image[..., ::-<NUM_LIT:1>])<EOL><DEDENT>imwrite(image, coords, '<STR_LIT>')<EOL>imwrite(result, coords_result, '<STR_LIT>')<EOL>

Example 4: Transforming coordinates using affine matrix.


model = onnx.load(onnx_input_path)<EOL>tf_rep = prepare(model)<EOL>img = np.load("<STR_LIT>")<EOL>output = tf_rep.run(img.reshape([<NUM_LIT:1>, <NUM_LIT>]))<EOL>print("<STR_LIT>", np.argmax(output))<EOL>

Reimplementation of the TensorFlow-onnx official tutorial convert the onnx file to specific: model

    Parameters
    -----------
    onnx_input_path : string
    the path where you save the onnx file.

    References
    -----------
    - `onnx-tf exporting tutorial <https://github.com/onnx/tutorials/blob/master/tutorials/OnnxTensorflowExport.ipynb>`__


def freeze_graph(graph_path, checkpoint_path, output_path, end_node_names, is_binary_graph):

_freeze_graph(<EOL>input_graph=graph_path, input_saver='<STR_LIT>', input_binary=is_binary_graph, input_checkpoint=checkpoint_path,<EOL>output_graph=output_path, output_node_names=end_node_names, restore_op_name='<STR_LIT>',<EOL>filename_tensor_name='<STR_LIT>', clear_devices=True, initializer_nodes=None<EOL>)<EOL>

Reimplementation of the TensorFlow official freeze_graph function to freeze the graph and checkpoint together:

    Parameters
    -----------
    graph_path : string
        the path where your graph file save.
    checkpoint_output_path : string
        the path where your checkpoint save.
    output_path : string
        the path where you want to save the output proto buff
    end_node_names : string
        the name of the end node in your graph you want to get in your proto buff
    is_binary_graph : boolean
        declare your file whether is a binary graph

    References
    ----------
    - `onnx-tf exporting tutorial <https://github.com/onnx/tutorials/blob/master/tutorials/OnnxTensorflowExport.ipynb>`__
    - `tensorflow freeze_graph <https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/freeze_graph.py>`


def google2_log_prefix(level, timestamp=None, file_and_line=None):

<EOL>global _level_names<EOL>now = timestamp or _time.time()<EOL>now_tuple = _time.localtime(now)<EOL>now_microsecond = int(<NUM_LIT> * (now % <NUM_LIT:1.0>))<EOL>(filename, line) = file_and_line or _GetFileAndLine()<EOL>basename = _os.path.basename(filename)<EOL>severity = '<STR_LIT:I>'<EOL>if level in _level_names:<EOL><INDENT>severity = _level_names[level][<NUM_LIT:0>]<EOL><DEDENT>s = '<STR_LIT>' % (<EOL>severity,<EOL>now_tuple[<NUM_LIT:1>],  <EOL>now_tuple[<NUM_LIT:2>],  <EOL>now_tuple[<NUM_LIT:3>],  <EOL>now_tuple[<NUM_LIT:4>],  <EOL>now_tuple[<NUM_LIT:5>],  <EOL>now_microsecond,<EOL>_get_thread_id(),<EOL>basename,<EOL>line<EOL>)<EOL>return s<EOL>

Assemble a logline prefix using the google2 format.


def log_if(level, msg, condition, *args):

if condition:<EOL><INDENT>vlog(level, msg, *args)<EOL><DEDENT>

Log 'msg % args' at level 'level' only if condition is fulfilled.


Sets the threshold for what messages will be logged.


<EOL>f = _sys._getframe()<EOL>our_file = f.f_code.co_filename<EOL>f = f.f_back<EOL>while f:<EOL><INDENT>code = f.f_code<EOL>if code.co_filename != our_file:<EOL><INDENT>return (code.co_filename, f.f_lineno)<EOL><DEDENT>f = f.f_back<EOL><DEDENT>return ('<STR_LIT>', <NUM_LIT:0>)<EOL>

Returns (filename, linenumber) for the stack frame.


if shape[<NUM_LIT:0>] != shape[<NUM_LIT:1>]:<EOL><INDENT>raise Exception('<STR_LIT>')<EOL><DEDENT>if shape[<NUM_LIT:3>] < shape[<NUM_LIT:2>]:<EOL><INDENT>raise Exception(<EOL>'<STR_LIT>'<EOL>)<EOL><DEDENT>filter_size = shape[<NUM_LIT:0>]<EOL>num_out_channels = shape[<NUM_LIT:2>]<EOL>num_in_channels = shape[<NUM_LIT:3>]<EOL>bilinear_kernel = np.zeros([filter_size, filter_size], dtype=np.float32)<EOL>scale_factor = (filter_size + <NUM_LIT:1>) // <NUM_LIT:2><EOL>if filter_size % <NUM_LIT:2> == <NUM_LIT:1>:<EOL><INDENT>center = scale_factor - <NUM_LIT:1><EOL><DEDENT>else:<EOL><INDENT>center = scale_factor - <NUM_LIT:0.5><EOL><DEDENT>for x in range(filter_size):<EOL><INDENT>for y in range(filter_size):<EOL><INDENT>bilinear_kernel[x, y] = (<NUM_LIT:1> - abs(x - center) / scale_factor) * (<NUM_LIT:1> - abs(y - center) / scale_factor)<EOL><DEDENT><DEDENT>weights = np.zeros((filter_size, filter_size, num_out_channels, num_in_channels))<EOL>for i in range(num_out_channels):<EOL><INDENT>weights[:, :, i, i] = bilinear_kernel<EOL><DEDENT>return tf.constant_initializer(value=weights, dtype=LayersConfig.tf_dtype)<EOL>

Returns the initializer that can be passed to DeConv2dLayer for initializing the
    weights in correspondence to channel-wise bilinear up-sampling.
    Used in segmentation approaches such as [FCN](https://arxiv.org/abs/1605.06211)

    Parameters
    ----------
    shape : tuple of int
        The shape of the filters, [height, width, output_channels, in_channels].
        It must match the shape passed to DeConv2dLayer.

    Returns
    -------
    ``tf.constant_initializer``
        A constant initializer with weights set to correspond to per channel bilinear upsampling
        when passed as W_int in DeConv2dLayer

    Examples
    --------
    - Upsampling by a factor of 2, ie e.g 100->200
    >>> import tensorflow as tf
    >>> import tensorlayer as tl
    >>> rescale_factor = 2
    >>> imsize = 128
    >>> num_channels = 3
    >>> filter_shape = (5, 5)
    >>> filter_size = (2 * rescale_factor - rescale_factor % 2) #Corresponding bilinear filter size
    >>> num_in_channels = 3
    >>> num_out_channels = 3
    >>> deconv_filter_shape = (filter_size, filter_size, num_out_channels, num_in_channels)
    >>> x = tf.placeholder(tf.float32, (1, imsize, imsize, num_channels))
    >>> net = tl.layers.InputLayer(x, name='input_layer')
    >>> bilinear_init = deconv2d_bilinear_upsampling_initializer(shape=filter_shape)
    >>> net = tl.layers.DeConv2dLayer(net,
    ...                    shape=filter_shape,
    ...                    output_shape=(1, imsize*rescale_factor, imsize*rescale_factor, num_out_channels),
    ...                    strides=(1, rescale_factor, rescale_factor, 1),
    ...                    W_init=bilinear_init,
    ...                    padding='SAME',
    ...                    act=None, name='g/h1/decon2d')


path = os.path.join(path, '<STR_LIT>')<EOL>logging.info("<STR_LIT>".format(path))<EOL>filename = '<STR_LIT>'<EOL>url = '<STR_LIT>'<EOL>maybe_download_and_extract(filename, path, url, extract=True)<EOL>data_path = os.path.join(path, '<STR_LIT>', '<STR_LIT:data>')<EOL>train_path = os.path.join(data_path, "<STR_LIT>")<EOL>valid_path = os.path.join(data_path, "<STR_LIT>")<EOL>test_path = os.path.join(data_path, "<STR_LIT>")<EOL>word_to_id = nlp.build_vocab(nlp.read_words(train_path))<EOL>train_data = nlp.words_to_word_ids(nlp.read_words(train_path), word_to_id)<EOL>valid_data = nlp.words_to_word_ids(nlp.read_words(valid_path), word_to_id)<EOL>test_data = nlp.words_to_word_ids(nlp.read_words(test_path), word_to_id)<EOL>vocab_size = len(word_to_id)<EOL>return train_data, valid_data, test_data, vocab_size<EOL>

Load Penn TreeBank (PTB) dataset.

    It is used in many LANGUAGE MODELING papers,
    including "Empirical Evaluation and Combination of Advanced Language
    Modeling Techniques", "Recurrent Neural Network Regularization".
    It consists of 929k training words, 73k validation words, and 82k test
    words. It has 10k words in its vocabulary.

    Parameters
    ----------
    path : str
        The path that the data is downloaded to, defaults is ``data/ptb/``.

    Returns
    --------
    train_data, valid_data, test_data : list of int
        The training, validating and testing data in integer format.
    vocab_size : int
        The vocabulary size.

    Examples
    --------
    >>> train_data, valid_data, test_data, vocab_size = tl.files.load_ptb_dataset()

    References
    ---------------
    - ``tensorflow.models.rnn.ptb import reader``
    - `Manual download <http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz>`__

    Notes
    ------
    - If you want to get the raw data, see the source code.


def load_flickr1M_dataset(tag='<STR_LIT>', size=<NUM_LIT:10>, path="<STR_LIT:data>", n_threads=<NUM_LIT:50>, printable=False):

import shutil<EOL>path = os.path.join(path, '<STR_LIT>')<EOL>logging.info("<STR_LIT>".format(size * <NUM_LIT:10>, size * <NUM_LIT>))<EOL>images_zip = [<EOL>'<STR_LIT>', '<STR_LIT>', '<STR_LIT>', '<STR_LIT>', '<STR_LIT>', '<STR_LIT>', '<STR_LIT>',<EOL>'<STR_LIT>', '<STR_LIT>', '<STR_LIT>'<EOL>]<EOL>tag_zip = '<STR_LIT>'<EOL>url = '<STR_LIT>'<EOL>for image_zip in images_zip[<NUM_LIT:0>:size]:<EOL><INDENT>image_folder = image_zip.split("<STR_LIT:.>")[<NUM_LIT:0>]<EOL>if folder_exists(os.path.join(path, image_folder)) is False:<EOL><INDENT>logging.info("<STR_LIT>".format(image_folder, path))<EOL>maybe_download_and_extract(image_zip, path, url, extract=True)<EOL>del_file(os.path.join(path, image_zip))<EOL>shutil.move(os.path.join(path, '<STR_LIT>'), os.path.join(path, image_folder))<EOL><DEDENT>else:<EOL><INDENT>logging.info("<STR_LIT>".format(image_folder, path))<EOL><DEDENT><DEDENT>if folder_exists(os.path.join(path, "<STR_LIT>")) is False:<EOL><INDENT>logging.info("<STR_LIT>".format(path))<EOL>maybe_download_and_extract(tag_zip, path, url, extract=True)<EOL>del_file(os.path.join(path, tag_zip))<EOL><DEDENT>else:<EOL><INDENT>logging.info("<STR_LIT>".format(path))<EOL><DEDENT>images_list = []<EOL>images_folder_list = []<EOL>for i in range(<NUM_LIT:0>, size):<EOL><INDENT>images_folder_list += load_folder_list(path=os.path.join(path, '<STR_LIT>' % i))<EOL><DEDENT>images_folder_list.sort(key=lambda s: int(s.split('<STR_LIT:/>')[-<NUM_LIT:1>]))  <EOL>for folder in images_folder_list[<NUM_LIT:0>:size * <NUM_LIT:10>]:<EOL><INDENT>tmp = load_file_list(path=folder, regx='<STR_LIT>', printable=False)<EOL>tmp.sort(key=lambda s: int(s.split('<STR_LIT:.>')[-<NUM_LIT:2>]))  <EOL>images_list.extend([os.path.join(folder, x) for x in tmp])<EOL><DEDENT>tag_list = []<EOL>tag_folder_list = load_folder_list(os.path.join(path, "<STR_LIT>"))<EOL>tag_folder_list.sort(key=lambda s: int(os.path.basename(s)))<EOL>for folder in tag_folder_list[<NUM_LIT:0>:size * <NUM_LIT:10>]:<EOL><INDENT>tmp = load_file_list(path=folder, regx='<STR_LIT>', printable=False)<EOL>tmp.sort(key=lambda s: int(s.split('<STR_LIT:.>')[-<NUM_LIT:2>]))  <EOL>tmp = [os.path.join(folder, s) for s in tmp]<EOL>tag_list += tmp<EOL><DEDENT>logging.info("<STR_LIT>".format(tag))<EOL>select_images_list = []<EOL>for idx, _val in enumerate(tag_list):<EOL><INDENT>tags = read_file(tag_list[idx]).split('<STR_LIT:\n>')<EOL>if tag in tags:<EOL><INDENT>select_images_list.append(images_list[idx])<EOL><DEDENT><DEDENT>logging.info("<STR_LIT>".format(tag))<EOL>images = visualize.read_images(select_images_list, '<STR_LIT>', n_threads=n_threads, printable=printable)<EOL>return images<EOL>

Load Flick1M dataset.

    Returns a list of images by a given tag from Flickr1M dataset,
    it will download Flickr1M from `the official website <http://press.liacs.nl/mirflickr/mirdownload.html>`__
    at the first time you use it.

    Parameters
    ------------
    tag : str or None
        What images to return.
            - If you want to get images with tag, use string like 'dog', 'red', see `Flickr Search <https://www.flickr.com/search/>`__.
            - If you want to get all images, set to ``None``.

    size : int
        integer between 1 to 10. 1 means 100k images ... 5 means 500k images, 10 means all 1 million images. Default is 10.
    path : str
        The path that the data is downloaded to, defaults is ``data/flickr25k/``.
    n_threads : int
        The number of thread to read image.
    printable : boolean
        Whether to print infomation when reading images, default is ``False``.

    Examples
    ----------
    Use 200k images

    >>> images = tl.files.load_flickr1M_dataset(tag='zebra', size=2)

    Use 1 Million images

    >>> images = tl.files.load_flickr1M_dataset(tag='zebra')


data_dir = '<STR_LIT>'<EOL>filename, drive_id = "<STR_LIT>", "<STR_LIT>"<EOL>save_path = os.path.join(path, filename)<EOL>image_path = os.path.join(path, data_dir)<EOL>if os.path.exists(image_path):<EOL><INDENT>logging.info('<STR_LIT>'.format(save_path))<EOL><DEDENT>else:<EOL><INDENT>exists_or_mkdir(path)<EOL>download_file_from_google_drive(drive_id, save_path)<EOL>zip_dir = '<STR_LIT>'<EOL>with zipfile.ZipFile(save_path) as zf:<EOL><INDENT>zip_dir = zf.namelist()[<NUM_LIT:0>]<EOL>zf.extractall(path)<EOL><DEDENT>os.remove(save_path)<EOL>os.rename(os.path.join(path, zip_dir), image_path)<EOL><DEDENT>data_files = load_file_list(path=image_path, regx='<STR_LIT>', printable=False)<EOL>for i, _v in enumerate(data_files):<EOL><INDENT>data_files[i] = os.path.join(image_path, data_files[i])<EOL><DEDENT>return data_files<EOL>

Load CelebA dataset

    Return a list of image path.

    Parameters
    -----------
    path : str
        The path that the data is downloaded to, defaults is ``data/celebA/``.