assert len(sentences) > <NUM_LIT:0><EOL>assert n > <NUM_LIT:0><EOL>words = _split_into_words(sentences)<EOL>return _get_ngrams(n, words)<EOL>

Calculates word n-grams for multiple sentences.


table = _lcs(x, y)<EOL>n, m = len(x), len(y)<EOL>return table[n, m]<EOL>

Returns the length of the Longest Common Subsequence between sequences x
and y.
Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence

Args:
  x: sequence of words
  y: sequence of words

Returns
  integer: Length of LCS between x and y


n, m = len(x), len(y)<EOL>table = dict()<EOL>for i in range(n + <NUM_LIT:1>):<EOL><INDENT>for j in range(m + <NUM_LIT:1>):<EOL><INDENT>if i == <NUM_LIT:0> or j == <NUM_LIT:0>:<EOL><INDENT>table[i, j] = <NUM_LIT:0><EOL><DEDENT>elif x[i - <NUM_LIT:1>] == y[j - <NUM_LIT:1>]:<EOL><INDENT>table[i, j] = table[i - <NUM_LIT:1>, j - <NUM_LIT:1>] + <NUM_LIT:1><EOL><DEDENT>else:<EOL><INDENT>table[i, j] = max(table[i - <NUM_LIT:1>, j], table[i, j - <NUM_LIT:1>])<EOL><DEDENT><DEDENT><DEDENT>return table<EOL>

Computes the length of the longest common subsequence (lcs) between two
strings. The implementation below uses a DP programming algorithm and runs
in O(nm) time where n = len(x) and m = len(y).
Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence

Args:
  x: collection of words
  y: collection of words

Returns:
  Table of dictionary of coord and len lcs


i, j = len(x), len(y)<EOL>table = _lcs(x, y)<EOL>def _recon(i, j):<EOL><INDENT>"""<STR_LIT>"""<EOL>if i == <NUM_LIT:0> or j == <NUM_LIT:0>:<EOL><INDENT>return []<EOL><DEDENT>elif x[i - <NUM_LIT:1>] == y[j - <NUM_LIT:1>]:<EOL><INDENT>return _recon(i - <NUM_LIT:1>, j - <NUM_LIT:1>) + [(x[i - <NUM_LIT:1>], i)]<EOL><DEDENT>elif table[i - <NUM_LIT:1>, j] > table[i, j - <NUM_LIT:1>]:<EOL><INDENT>return _recon(i - <NUM_LIT:1>, j)<EOL><DEDENT>else:<EOL><INDENT>return _recon(i, j - <NUM_LIT:1>)<EOL><DEDENT><DEDENT>recon_tuple = tuple(map(lambda x: x[<NUM_LIT:0>], _recon(i, j)))<EOL>return recon_tuple<EOL>

Returns the Longest Subsequence between x and y.
Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence

Args:
  x: sequence of words
  y: sequence of words

Returns:
  sequence: LCS of x and y


def multi_rouge_n(sequences, scores_ids, n=<NUM_LIT:2>):

ngrams = [_get_word_ngrams(n, sequence) for sequence in sequences]<EOL>counts = [len(ngram) for ngram in ngrams]<EOL>scores = []<EOL>for hyp_id, ref_id in scores_ids:<EOL><INDENT>evaluated_ngrams = ngrams[hyp_id]<EOL>evaluated_count = counts[hyp_id]<EOL>reference_ngrams = ngrams[ref_id]<EOL>reference_count = counts[ref_id]<EOL>overlapping_ngrams = evaluated_ngrams.intersection(reference_ngrams)<EOL>overlapping_count = len(overlapping_ngrams)<EOL>scores += [f_r_p_rouge_n(evaluated_count,<EOL>reference_count, overlapping_count)]<EOL><DEDENT>return scores<EOL>

Efficient way to compute highly repetitive scoring
i.e. sequences are involved multiple time

Args:
    sequences(list[str]): list of sequences (either hyp or ref)
    scores_ids(list[tuple(int)]): list of pairs (hyp_id, ref_id)
        ie. scores[i] = rouge_n(scores_ids[i][0],
                                scores_ids[i][1])

Returns:
    scores: list of length `len(scores_ids)` containing rouge `n`
            scores as a dict with 'f', 'r', 'p'
Raises:
    KeyError: if there's a value of i in scores_ids that is not in
              [0, len(sequences)[


def rouge_n(evaluated_sentences, reference_sentences, n=<NUM_LIT:2>):

if len(evaluated_sentences) <= <NUM_LIT:0> or len(reference_sentences) <= <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>evaluated_ngrams = _get_word_ngrams(n, evaluated_sentences)<EOL>reference_ngrams = _get_word_ngrams(n, reference_sentences)<EOL>reference_count = len(reference_ngrams)<EOL>evaluated_count = len(evaluated_ngrams)<EOL>overlapping_ngrams = evaluated_ngrams.intersection(reference_ngrams)<EOL>overlapping_count = len(overlapping_ngrams)<EOL>return f_r_p_rouge_n(evaluated_count, reference_count, overlapping_count)<EOL>

Computes ROUGE-N of two text collections of sentences.
Sourece: http://research.microsoft.com/en-us/um/people/cyl/download/
papers/rouge-working-note-v1.3.1.pdf

Args:
  evaluated_sentences: The sentences that have been picked by the
                       summarizer
  reference_sentences: The sentences from the referene set
  n: Size of ngram.  Defaults to 2.

Returns:
  A tuple (f1, precision, recall) for ROUGE-N

Raises:
  ValueError: raises exception if a param has len <= 0


def _union_lcs(evaluated_sentences, reference_sentence, prev_union=None):

if prev_union is None:<EOL><INDENT>prev_union = set()<EOL><DEDENT>if len(evaluated_sentences) <= <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>lcs_union = prev_union<EOL>prev_count = len(prev_union)<EOL>reference_words = _split_into_words([reference_sentence])<EOL>combined_lcs_length = <NUM_LIT:0><EOL>for eval_s in evaluated_sentences:<EOL><INDENT>evaluated_words = _split_into_words([eval_s])<EOL>lcs = set(_recon_lcs(reference_words, evaluated_words))<EOL>combined_lcs_length += len(lcs)<EOL>lcs_union = lcs_union.union(lcs)<EOL><DEDENT>new_lcs_count = len(lcs_union) - prev_count<EOL>return new_lcs_count, lcs_union<EOL>

Returns LCS_u(r_i, C) which is the LCS score of the union longest common
subsequence between reference sentence ri and candidate summary C.
For example:
if r_i= w1 w2 w3 w4 w5, and C contains two sentences: c1 = w1 w2 w6 w7 w8
and c2 = w1 w3 w8 w9 w5, then the longest common subsequence of r_i and c1
is "w1 w2" and the longest common subsequence of r_i and c2 is "w1 w3 w5".
The union longest common subsequence of r_i, c1, and c2 is "w1 w2 w3 w5"
and LCS_u(r_i, C) = 4/5.

Args:
  evaluated_sentences: The sentences that have been picked by the
                       summarizer
  reference_sentence: One of the sentences in the reference summaries

Returns:
  float: LCS_u(r_i, C)

ValueError:
  Raises exception if a param has len <= 0


if len(evaluated_sentences) <= <NUM_LIT:0> or len(reference_sentences) <= <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>m = len(set(_split_into_words(reference_sentences)))<EOL>n = len(set(_split_into_words(evaluated_sentences)))<EOL>union_lcs_sum_across_all_references = <NUM_LIT:0><EOL>union = set()<EOL>for ref_s in reference_sentences:<EOL><INDENT>lcs_count, union = _union_lcs(evaluated_sentences,<EOL>ref_s,<EOL>prev_union=union)<EOL>union_lcs_sum_across_all_references += lcs_count<EOL><DEDENT>llcs = union_lcs_sum_across_all_references<EOL>r_lcs = llcs / m<EOL>p_lcs = llcs / n<EOL>beta = p_lcs / (r_lcs + <NUM_LIT>)<EOL>num = (<NUM_LIT:1> + (beta**<NUM_LIT:2>)) * r_lcs * p_lcs<EOL>denom = r_lcs + ((beta**<NUM_LIT:2>) * p_lcs)<EOL>f_lcs = num / (denom + <NUM_LIT>)<EOL>return {"<STR_LIT:f>": f_lcs, "<STR_LIT:p>": p_lcs, "<STR_LIT:r>": r_lcs}<EOL>

Computes ROUGE-L (summary level) of two text collections of sentences.
http://research.microsoft.com/en-us/um/people/cyl/download/papers/
rouge-working-note-v1.3.1.pdf

Calculated according to:
R_lcs = SUM(1, u)[LCS<union>(r_i,C)]/m
P_lcs = SUM(1, u)[LCS<union>(r_i,C)]/n
F_lcs = ((1 + beta^2)*R_lcs*P_lcs) / (R_lcs + (beta^2) * P_lcs)

where:
SUM(i,u) = SUM from i through u
u = number of sentences in reference summary
C = Candidate summary made up of v sentences
m = number of words in reference summary
n = number of words in candidate summary

Args:
  evaluated_sentences: The sentences that have been picked by the
                       summarizer
  reference_sentence: One of the sentences in the reference summaries

Returns:
  A float: F_lcs

Raises:
  ValueError: raises exception if a param has len <= 0


try:<EOL><INDENT>return codecs.decode(text, '<STR_LIT:utf-8>')<EOL><DEDENT>except:<EOL><INDENT>return text<EOL><DEDENT>

Decodes the text from utf-8 format.

    Parameters
    ----------
    text : str
        Text to be decoded

    Returns
    -------
    str
        Decoded text


try:<EOL><INDENT>return codecs.encode(text, '<STR_LIT:utf-8>')<EOL><DEDENT>except:<EOL><INDENT>return text<EOL><DEDENT>

Encodes the text to utf-8 format

    Parameters
    ----------
    text : str
        Text to be encoded

    Returns
    -------
    str
        Encoded text


def animate(frames, interval, name, iterations=<NUM_LIT:2>):

for i in range(iterations):<EOL><INDENT>for frame in frames:<EOL><INDENT>frame = get_coded_text(frame)<EOL>output = "<STR_LIT>".format(frame, name)<EOL>sys.stdout.write(output)<EOL>sys.stdout.write(CLEAR_LINE)<EOL>sys.stdout.flush()<EOL>time.sleep(<NUM_LIT> * interval)<EOL><DEDENT><DEDENT>

Animate given frame for set number of iterations.

    Parameters
    ----------
    frames : list
        Frames for animating
    interval : float
        Interval between two frames
    name : str
        Name of animation
    iterations : int, optional
        Number of loops for animations


<EOL>SigningCertURLValidator(certificate_url_regex).validate(message)<EOL>if not isinstance(max_age, datetime.timedelta):<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>MessageAgeValidator(max_age).validate(message)<EOL>certificate = get_certificate(message["<STR_LIT>"])<EOL>SignatureValidator(certificate).validate(message)<EOL>

Validate a decoded SNS message.

Parameters:
    message:
        Decoded SNS message.

    get_certificate:
        Function that receives a URL, and returns the certificate from that
        URL as a string. The default doesn't implement caching.

    certificate_url_regex:
        Regex that validates the signing certificate URL. Default value
        checks it's hosted on an AWS-controlled domain, in the format
        "https://sns.<data-center>.amazonaws.com/"

    max_age:
        Maximum age of an SNS message before it fails validation, expressed
        as a `datetime.timedelta`. Defaults to one hour, the max. lifetime
        of an SNS message.


cls.directory = tempfile.mkdtemp()<EOL>cls.module_path = os.path.join(cls.directory, '<STR_LIT>')<EOL>os.mkdir(cls.module_path)<EOL>with open(os.path.join(cls.module_path, '<STR_LIT>'), '<STR_LIT:w>') as f:<EOL><INDENT>f.write(INIT_PY)<EOL><DEDENT>with open(os.path.join(cls.module_path, '<STR_LIT>'), '<STR_LIT:w>') as f:<EOL><INDENT>f.write(TEST_CLASSES_PY)<EOL><DEDENT>embedded_module = os.path.join(cls.module_path, '<STR_LIT>')<EOL>os.mkdir(embedded_module)<EOL>with open(os.path.join(embedded_module, '<STR_LIT>'), '<STR_LIT:w>') as f:<EOL><INDENT>f.write('<STR_LIT:\n>')<EOL><DEDENT>with open(os.path.join(embedded_module, '<STR_LIT>'), '<STR_LIT:w>') as f:<EOL><INDENT>f.write('<STR_LIT:\n>')<EOL><DEDENT>sub_module = os.path.join(embedded_module, '<STR_LIT>')<EOL>os.mkdir(sub_module)<EOL>with open(os.path.join(sub_module, '<STR_LIT>'), '<STR_LIT:w>') as f:<EOL><INDENT>f.write('<STR_LIT:\n>')<EOL><DEDENT>with open(os.path.join(sub_module, '<STR_LIT>'), '<STR_LIT:w>') as f:<EOL><INDENT>f.write(TEST_CLASSES_PY)<EOL><DEDENT>garbage_folder = os.path.join(cls.module_path, '<STR_LIT>')<EOL>os.mkdir(garbage_folder)<EOL>cls.plugin_manager = pynsive.PluginManager()<EOL>cls.plugin_manager.plug_into(cls.directory)<EOL>

This crazy setUp method for the following unit tests creates a
temporary plugin directory and then drops a Python module and related
testing code into it. The method then spins up a plugin context via
a pynsive.PluginManager instance.


Removes the meta_path hook associated with this manager.


[self.finder.add_path(path) for path in paths]<EOL>

Adds all arguments passed as plugin directories to search when loading
modules.


self.msg = msg<EOL>

Creates a loader error.

:param msg: the loader error message to relay upwards


def __init__(self, module_path, module_name, load_target, is_pkg):

self.module_path = module_path<EOL>self.module_name = module_name<EOL>self.load_target = load_target<EOL>self.is_pkg = is_pkg<EOL>

Creates a new module loader.

:param module_path: the path where the module resides
:param module_name: the full name of the module to load
:param load_target: the absolute path of the module file to load
:param is_pkg: true if this module is a package, false otherwise


if module_name != self.module_name:<EOL><INDENT>raise LoaderError(<EOL>'<STR_LIT>')<EOL><DEDENT>if module_name in sys.modules:<EOL><INDENT>return sys.modules[module_name]<EOL><DEDENT>module = self.load_module_py_path(module_name, self.load_target)<EOL>if self.is_pkg:<EOL><INDENT>module.__path__ = [self.module_path]<EOL>module.__package__ = module_name<EOL><DEDENT>else:<EOL><INDENT>module.__package__ = module_name.rpartition('<STR_LIT:.>')[<NUM_LIT:0>]<EOL><DEDENT>sys.modules[module_name] = module<EOL>return module<EOL>

Loads a module's code and sets the module's expected hidden
variables. For more information on these variables and what they
are for, please see PEP302.

:param module_name: the full name of the module to load


self.paths = paths if paths else list()<EOL>

Creates a module finder.

:param paths: the paths to include in the search list by default


if path not in self.paths:<EOL><INDENT>self.paths.append(path)<EOL><DEDENT>

Adds a path to search through when attempting to look up a module.

:param path: the path the add to the list of searchable paths


def find_module(self, module_name, path=None):

module_path = os.path.join(*module_name.split(MODULE_PATH_SEP))<EOL>for search_root in self.paths:<EOL><INDENT>target_path = os.path.join(search_root, module_path)<EOL>is_pkg = False<EOL>if os.path.isdir(target_path):<EOL><INDENT>target_file = os.path.join(target_path, '<STR_LIT>')<EOL>is_pkg = True<EOL><DEDENT>else:<EOL><INDENT>target_file = '<STR_LIT>'.format(target_path)<EOL><DEDENT>if os.path.exists(target_file):<EOL><INDENT>return ModuleLoader(<EOL>target_path, module_name, target_file, is_pkg)<EOL><DEDENT><DEDENT>return None<EOL>

Searches the paths for the required module.

:param module_name: the full name of the module to find
:param path: set to None when the module in being searched for is a
             top-level module - otherwise this is set to
             package.__path__ for submodules and subpackages (unused)


return importlib.import_module(module_name)<EOL>


return hasattr(module, PATH_ATTRUBITE) and len(module.__path__) > <NUM_LIT:0><EOL>

Checks to make sure that the passed module has the correct hidden
variables set.


found = list()<EOL>if os.path.isdir(directory):<EOL><INDENT>for entry in os.listdir(directory):<EOL><INDENT>next_dir = os.path.join(directory, entry)<EOL>if os.path.isfile(os.path.join(next_dir, MODULE_INIT_FILE)):<EOL><INDENT>found.append(entry)<EOL><DEDENT><DEDENT><DEDENT>return found<EOL>

Attempts to list all of the modules and submodules found within a given
directory tree. This function searches the top-level of the directory
tree for potential python modules and returns a list of candidate names.

**Note:** This function returns a list of strings representing
discovered module names, not the actual, loaded modules.

:param directory: the directory to search for modules.


found = list()<EOL>if os.path.isdir(directory):<EOL><INDENT>for entry in os.listdir(directory):<EOL><INDENT>next_dir = os.path.join(directory, entry)<EOL>if os.path.isfile(os.path.join(next_dir, MODULE_INIT_FILE)):<EOL><INDENT>modules = _search_for_modules(next_dir, True, entry)<EOL>found.extend(modules)<EOL><DEDENT><DEDENT><DEDENT>return found<EOL>

Attempts to list all of the modules and submodules found within a given
directory tree. This function recursively searches the directory tree
for potential python modules and returns a list of candidate names.

**Note:** This function returns a list of strings representing
discovered module names, not the actual, loaded modules.

:param directory: the directory to search for modules.


module = import_module(mname)<EOL>if not module:<EOL><INDENT>raise ImportError('<STR_LIT>'.format(mname))<EOL><DEDENT>found = list()<EOL>if _should_use_module_path(module):<EOL><INDENT>mpath = module.__path__[<NUM_LIT:0>]<EOL><DEDENT>else:<EOL><INDENT>mpaths = sys.path<EOL>mpath = _scan_paths_for(mname, mpaths)<EOL><DEDENT>if mpath:<EOL><INDENT>for pmname in _search_for_modules(mpath):<EOL><INDENT>found_mod = MODULE_PATH_SEP.join((mname, pmname))<EOL>found.append(found_mod)<EOL><DEDENT><DEDENT>return found<EOL>

Attempts to list all of the submodules under a module. This function
works for modules located in the default path as well as extended paths
via the sys.meta_path hooks.

This function carries the expectation that the hidden module variable
'__path__' has been set correctly.

:param mname: the module name to descend into


module = import_module(mname)<EOL>if not module:<EOL><INDENT>raise ImportError('<STR_LIT>'.format(mname))<EOL><DEDENT>found = list()<EOL>if _should_use_module_path(module):<EOL><INDENT>mpath = module.__path__[<NUM_LIT:0>]<EOL><DEDENT>else:<EOL><INDENT>mpaths = sys.path<EOL>mpath = _scan_paths_for(mname, mpaths)<EOL><DEDENT>if mpath:<EOL><INDENT>for pmname in _search_for_modules(mpath, recursive=True):<EOL><INDENT>found_mod = MODULE_PATH_SEP.join((mname, pmname))<EOL>found.append(found_mod)<EOL><DEDENT><DEDENT>return found<EOL>

Attempts to the submodules under a module recursively. This function
works for modules located in the default path as well as extended paths
via the sys.meta_path hooks.

This function carries the expectation that the hidden module variable
'__path__' has been set correctly.

:param mname: the module name to descend into


found = list()<EOL>module = import_module(mname)<EOL>if inspect.ismodule(module):<EOL><INDENT>[found.append(mod) for mod in _list_classes(module, cls_filter)]<EOL><DEDENT>return found<EOL>

Attempts to list all of the classes within a specified module. This
function works for modules located in the default path as well as
extended paths via the sys.meta_path hooks.

If a class filter is set, it will be called with each class as its
parameter. This filter's return value must be interpretable as a
boolean. Results that evaluate as True will include the type in the
list of returned classes. Results that evaluate as False will exclude
the type in the list of returned classes.

:param mname: of the module to descend into
:param cls_filter: a function to call to determine what classes should be
            included.


found = list()<EOL>mnames = rlist_modules(module)<EOL>for mname in mnames:<EOL><INDENT>[found.append(c) for c in list_classes(mname, cls_filter)]<EOL><DEDENT>return found<EOL>

Attempts to list all of the classes within a given module namespace.
This method, unlike list_classes, will recurse into discovered
submodules.

If a type filter is set, it will be called with each class as its
parameter. This filter's return value must be interpretable as a
boolean. Results that evaluate as True will include the type in the
list of returned classes. Results that evaluate as False will exclude
the type in the list of returned classes.

:param mname: of the module to descend into
:param cls_filter: a function to call to determine what classes should be
            included.


def apk(actual, predicted, k=<NUM_LIT:10>):

if len(predicted)>k:<EOL><INDENT>predicted = predicted[:k]<EOL><DEDENT>score = <NUM_LIT:0.0><EOL>num_hits = <NUM_LIT:0.0><EOL>for i,p in enumerate(predicted):<EOL><INDENT>if p in actual and p not in predicted[:i]:<EOL><INDENT>num_hits += <NUM_LIT:1.0><EOL>score += num_hits / (i+<NUM_LIT:1.0>)<EOL><DEDENT><DEDENT>if len(actual) == <NUM_LIT:0>:<EOL><INDENT>return <NUM_LIT:0.0><EOL><DEDENT>return score / min(len(actual), k)<EOL>

Computes the average precision at k.

This function computes the average prescision at k between two lists of
items.

Parameters
----------
actual : list
         A list of elements that are to be predicted (order doesn't matter)
predicted : list
            A list of predicted elements (order does matter)
k : int, optional
    The maximum number of predicted elements

Returns
-------
score : double
        The average precision at k over the input lists


def mapk(actual, predicted, k=<NUM_LIT:10>):

return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])<EOL>

Computes the mean average precision at k.

This function computes the mean average prescision at k between two lists
of lists of items.

Parameters
----------
actual : list
         A list of lists of elements that are to be predicted
         (order doesn't matter in the lists)
predicted : list
            A list of lists of predicted elements
            (order matters in the lists)
k : int, optional
    The maximum number of predicted elements

Returns
-------
score : double
        The mean average precision at k over the input lists


return struct.unpack('<STR_LIT>', hashlib.sha1(data).digest()[:<NUM_LIT:4>])[<NUM_LIT:0>]<EOL>

A 32-bit hash function based on SHA1.

    Args:
        data (bytes): the data to generate 32-bit integer hash from.

    Returns:
        int: an integer hash value that can be encoded using 32 bits.


return struct.unpack('<STR_LIT>', hashlib.sha1(data).digest()[:<NUM_LIT:8>])[<NUM_LIT:0>]<EOL>

A 32-bit hash function based on SHA1.

    Args:
        data (bytes): the data to generate 64-bit integer hash from.

    Returns:
        int: an integer hash value that can be encoded using 64 bits.


<EOL>hv = self.hashfunc(b)<EOL>reg_index = hv & (self.m - <NUM_LIT:1>)<EOL>bits = hv >> self.p<EOL>self.reg[reg_index] = max(self.reg[reg_index], self._get_rank(bits))<EOL>

Update the HyperLogLog with a new data value in bytes.
The value will be hashed using the hash function specified by
the `hashfunc` argument in the constructor.

Args:
    b: The value to be hashed using the hash function specified.

Example:
    To update with a new string value (using the default SHA1 hash
    function, which requires bytes as input):

    .. code-block:: python

        hll = HyperLogLog()
        hll.update("new value".encode('utf-8'))

    We can also use a different hash function, for example, `pyfarmhash`:

    .. code-block:: python

        import farmhash
        def _hash_32(b):
            return farmhash.hash32(b)
        hll = HyperLogLog(hashfunc=_hash_32)
        hll.update("new value")


<EOL>e = self.alpha * float(self.m ** <NUM_LIT:2>) / np.sum(<NUM_LIT>**(-self.reg))<EOL>if e <= (<NUM_LIT> / <NUM_LIT>) * self.m:<EOL><INDENT>num_zero = self.m - np.count_nonzero(self.reg)<EOL>return self._linearcounting(num_zero)<EOL><DEDENT>if e <= (<NUM_LIT:1.0> / <NUM_LIT>) * (<NUM_LIT:1> << <NUM_LIT:32>):<EOL><INDENT>return e<EOL><DEDENT>return self._largerange_correction(e)<EOL>

Estimate the cardinality of the data values seen so far.

Returns:
    int: The estimated cardinality.


if self.m != other.m or self.p != other.p:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>self.reg = np.maximum(self.reg, other.reg)<EOL>

Merge the other HyperLogLog with this one, making this the union of the
two.

Args:
    other (datasketch.HyperLogLog):


return copy.copy(self.reg)<EOL>

Returns:
    numpy.array: The current internal state.


return HyperLogLog(reg=self.digest())<EOL>

Create a copy of the current HyperLogLog by exporting its state.

Returns:
    datasketch.HyperLogLog:


if np.any(self.reg):<EOL><INDENT>return False<EOL><DEDENT>return True<EOL>

Returns:
    bool: True if the current HyperLogLog is empty - at the state of just
    initialized.


self.reg = np.zeros((self.m,), dtype=np.int8)<EOL>


return len(self.reg)<EOL>

Returns:
    int: Get the size of the HyperLogLog as the size of
        `reg`.


return type(self) is type(other) andself.p == other.p andself.m == other.m andnp.array_equal(self.reg, other.reg)<EOL>

Check equivalence between two HyperLogLogs

Args:
    other (datasketch.HyperLogLog):

Returns:
    bool: True if both have the same internal state.


if other.seed != self.seed:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if len(self) != len(other):<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>intersection = <NUM_LIT:0><EOL>for this, that in zip(self.hashvalues, other.hashvalues):<EOL><INDENT>if np.array_equal(this, that):<EOL><INDENT>intersection += <NUM_LIT:1><EOL><DEDENT><DEDENT>return float(intersection) / float(len(self))<EOL>

Estimate the `weighted Jaccard similarity`_ between the
        multi-sets represented by this weighted MinHash and the other.

        Args:
            other (datasketch.WeightedMinHash): The other weighted MinHash.

        Returns:
            float: The weighted Jaccard similarity between 0.0 and 1.0.

        .. _`weighted Jaccard similarity`: http://mathoverflow.net/questions/123339/weighted-jaccard-similarity


return copy.copy(self.hashvalues)<EOL>

Export the hash values, which is the internal state of the
        weighted MinHash.

        Returns:
            numpy.array: The hash values which is a Numpy array.


return WeightedMinHash(self.seed, self.digest())<EOL>

Returns:
    datasketch.WeightedMinHash: A copy of this weighted MinHash by exporting
    its state.


return len(self.hashvalues)<EOL>

Returns:
    int: The number of hash values.


return type(self) is type(other) andself.seed == other.seed andnp.array_equal(self.hashvalues, other.hashvalues)<EOL>

Returns:
    bool: If their seeds and hash values are both equal then two
    are equivalent.


if not isinstance(v, collections.Iterable):<EOL><INDENT>raise TypeError("<STR_LIT>")<EOL><DEDENT>if not len(v) == self.dim:<EOL><INDENT>raise ValueError("<STR_LIT>" % self.dim)<EOL><DEDENT>if not isinstance(v, np.ndarray):<EOL><INDENT>v = np.array(v, dtype=np.float32)<EOL><DEDENT>elif v.dtype != np.float32:<EOL><INDENT>v = v.astype(np.float32)<EOL><DEDENT>hashvalues = np.zeros((self.sample_size, <NUM_LIT:2>), dtype=np.int)<EOL>vzeros = (v == <NUM_LIT:0>)<EOL>if vzeros.all():<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>v[vzeros] = np.nan<EOL>vlog = np.log(v)<EOL>for i in range(self.sample_size):<EOL><INDENT>t = np.floor((vlog / self.rs[i]) + self.betas[i])<EOL>ln_y = (t - self.betas[i]) * self.rs[i]<EOL>ln_a = self.ln_cs[i] - ln_y - self.rs[i]<EOL>k = np.nanargmin(ln_a)<EOL>hashvalues[i][<NUM_LIT:0>], hashvalues[i][<NUM_LIT:1>] = k, int(t[k])<EOL><DEDENT>return WeightedMinHash(self.seed, hashvalues)<EOL>

Create a new weighted MinHash given a weighted Jaccard vector.
        Each dimension is an integer
        frequency of the corresponding element in the multi-set represented
        by the vector.

        Args:
            v (numpy.array): The Jaccard vector.


def _optimal_param(threshold, num_perm, false_positive_weight,<EOL>false_negative_weight):

min_error = float("<STR_LIT>")<EOL>opt = (<NUM_LIT:0>, <NUM_LIT:0>)<EOL>for b in range(<NUM_LIT:1>, num_perm+<NUM_LIT:1>):<EOL><INDENT>max_r = int(num_perm / b)<EOL>for r in range(<NUM_LIT:1>, max_r+<NUM_LIT:1>):<EOL><INDENT>fp = _false_positive_probability(threshold, b, r)<EOL>fn = _false_negative_probability(threshold, b, r)<EOL>error = fp*false_positive_weight + fn*false_negative_weight<EOL>if error < min_error:<EOL><INDENT>min_error = error<EOL>opt = (b, r)<EOL><DEDENT><DEDENT><DEDENT>return opt<EOL>

Compute the optimal `MinHashLSH` parameter that minimizes the weighted sum
of probabilities of false positive and false negative.


def insert(self, key, minhash, check_duplication=True):

self._insert(key, minhash, check_duplication=check_duplication, buffer=False)<EOL>

Insert a key to the index, together
with a MinHash (or weighted MinHash) of the set referenced by
the key.

:param str key: The identifier of the set.
:param datasketch.MinHash minhash: The MinHash of the set.
:param bool check_duplication: To avoid duplicate keys in the storage (`default=True`).
                               It's recommended to not change the default, but
                               if you want to avoid the overhead during insert
                               you can set `check_duplication = False`.


return MinHashLSHInsertionSession(self, buffer_size=buffer_size)<EOL>

Create a context manager for fast insertion into this index.

:param int buffer_size: The buffer size for insert_session mode (default=50000).

Returns:
    datasketch.lsh.MinHashLSHInsertionSession


if len(minhash) != self.h:<EOL><INDENT>raise ValueError("<STR_LIT>"<EOL>% (self.h, len(minhash)))<EOL><DEDENT>candidates = set()<EOL>for (start, end), hashtable in zip(self.hashranges, self.hashtables):<EOL><INDENT>H = self._H(minhash.hashvalues[start:end])<EOL>for key in hashtable.get(H):<EOL><INDENT>candidates.add(key)<EOL><DEDENT><DEDENT>if self.prepickle:<EOL><INDENT>return [pickle.loads(key) for key in candidates]<EOL><DEDENT>else:<EOL><INDENT>return list(candidates)<EOL><DEDENT>

Giving the MinHash of the query set, retrieve
the keys that references sets with Jaccard
similarities greater than the threshold.

Args:
    minhash (datasketch.MinHash): The MinHash of the query set.

Returns:
    `list` of unique keys.


if self.prepickle:<EOL><INDENT>key = pickle.dumps(key)<EOL><DEDENT>return key in self.keys<EOL>

Args:
    key (hashable): The unique identifier of a set.

Returns:
    bool: True only if the key exists in the index.


if self.prepickle:<EOL><INDENT>key = pickle.dumps(key)<EOL><DEDENT>if key not in self.keys:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>for H, hashtable in zip(self.keys[key], self.hashtables):<EOL><INDENT>hashtable.remove_val(H, key)<EOL>if not hashtable.get(H):<EOL><INDENT>hashtable.remove(H)<EOL><DEDENT><DEDENT>self.keys.remove(key)<EOL>

Remove the key from the index.

Args:
    key (hashable): The unique identifier of a set.


return any(t.size() == <NUM_LIT:0> for t in self.hashtables)<EOL>

Returns:
    bool: Check if the index is empty.


counts = [<EOL>hashtable.itemcounts() for hashtable in self.hashtables]<EOL>return counts<EOL>

Returns a list of length ``self.b`` with elements representing the
number of keys stored under each bucket for the given permutation.


if self.prepickle:<EOL><INDENT>key_set = [pickle.dumps(key) for key in set(keys)]<EOL><DEDENT>else:<EOL><INDENT>key_set = list(set(keys))<EOL><DEDENT>hashtables = [unordered_storage({'<STR_LIT:type>': '<STR_LIT>'}) for _ in<EOL>range(self.b)]<EOL>Hss = self.keys.getmany(*key_set)<EOL>for key, Hs in zip(key_set, Hss):<EOL><INDENT>for H, hashtable in zip(Hs, hashtables):<EOL><INDENT>hashtable.insert(H, key)<EOL><DEDENT><DEDENT>return [hashtable.itemcounts() for hashtable in hashtables]<EOL>

Returns the bucket allocation counts (see :func:`~datasketch.MinHashLSH.get_counts` above)
restricted to the list of keys given.

Args:
    keys (hashable) : the keys for which to get the bucket allocation
        counts


self.lsh._insert(key, minhash, check_duplication=check_duplication,<EOL>buffer=True)<EOL>

Insert a unique key to the index, together
with a MinHash (or weighted MinHash) of the set referenced by
the key.

Args:
    key (hashable): The unique identifier of the set.
    minhash (datasketch.MinHash): The MinHash of the set.


async with self._lock:<EOL><INDENT>for t in self.hashtables:<EOL><INDENT>await t.close()<EOL><DEDENT>if self.keys is not None:<EOL><INDENT>await self.keys.close()<EOL><DEDENT>self._initialized = False<EOL><DEDENT>

Cleanup client resources and disconnect from AsyncMinHashLSH storage.


async def insert(self, key, minhash, check_duplication=True):

await self._insert(key, minhash, check_duplication=check_duplication, buffer=False)<EOL>


return AsyncMinHashLSHInsertionSession(self, batch_size=batch_size)<EOL>

Create a asynchronous context manager for fast insertion in index.

:param int batch_size: the size of chunks to use in insert_session mode (default=10000).

:return: datasketch.experimental.aio.lsh.AsyncMinHashLSHSession

Example:
    .. code-block:: python

        from datasketch.experimental.aio.lsh import AsyncMinHashLSH
        from datasketch import MinHash

        def chunk(it, size):
            it = iter(it)
            return iter(lambda: tuple(islice(it, size)), ())

        _chunked_str = chunk((random.choice(string.ascii_lowercase) for _ in range(10000)), 4)
        seq = frozenset(chain((''.join(s) for s in _chunked_str), ('aahhb', 'aahh', 'aahhc', 'aac', 'kld', 'bhg', 'kkd', 'yow', 'ppi', 'eer')))
        objs = [MinHash(16) for _ in range(len(seq))]
        for e, obj in zip(seq, objs):
            for i in e:
                obj.update(i.encode('utf-8'))
        data = [(e, m) for e, m in zip(seq, objs)]

        _storage_config_redis = {'type': 'aiomongo', 'mongo': {'host': 'localhost', 'port': 27017}}
        async def func():
            async with AsyncMinHashLSH(storage_config=_storage_config_redis, threshold=0.5, num_perm=16) as lsh:
                async with lsh.insertion_session(batch_size=1000) as session:
                    fs = (session.insert(key, minhash, check_duplication=True) for key, minhash in data)
                    await asyncio.gather(*fs)


return AsyncMinHashLSHDeleteSession(self, batch_size=batch_size)<EOL>

Create a asynchronous context manager for fast removal of keys
from index.

:param int batch_size: the size of chunks to use in insert_session mode (default=10000).

:return: datasketch.experimental.aio.lsh.AsyncMinHashLSHSession

Example:
    .. code-block:: python

        from datasketch.experimental.aio.lsh import AsyncMinHashLSH
        from datasketch import MinHash

        def chunk(it, size):
            it = iter(it)
            return iter(lambda: tuple(islice(it, size)), ())

        _chunked_str = chunk((random.choice(string.ascii_lowercase) for _ in range(10000)), 4)
        seq = frozenset(chain((''.join(s) for s in _chunked_str), ('aahhb', 'aahh', 'aahhc', 'aac', 'kld', 'bhg', 'kkd', 'yow', 'ppi', 'eer')))
        objs = [MinHash(16) for _ in range(len(seq))]
        for e, obj in zip(seq, objs):
            for i in e:
                obj.update(i.encode('utf-8'))
        data = [(e, m) for e, m in zip(seq, objs)]

        _storage_config_redis = {'type': 'aiomongo', 'mongo': {'host': 'localhost', 'port': 27017}}
        async def func():
            async with AsyncMinHashLSH(storage_config=_storage_config_redis, threshold=0.5, num_perm=16) as lsh:
                async with lsh.insertion_session(batch_size=1000) as session:
                    fs = (session.insert(key, minhash, check_duplication=True) for key, minhash in data)
                    await asyncio.gather(*fs)

                async with lsh.delete_session(batch_size=3) as session:
                    fs = (session.remove(key) for key in keys_to_remove)
                    await asyncio.gather(*fs)


if len(minhash) != self.h:<EOL><INDENT>raise ValueError("<STR_LIT>"<EOL>"<STR_LIT>" % (self.h, len(minhash)))<EOL><DEDENT>fs = (hashtable.get(self._H(minhash.hashvalues[start:end]))<EOL>for (start, end), hashtable in zip(self.hashranges, self.hashtables))<EOL>candidates = frozenset(chain.from_iterable(await asyncio.gather(*fs)))<EOL>return list(candidates)<EOL>


return await self.keys.has_key(key)<EOL>


await self._remove(key, buffer=False)<EOL>


for t in self.hashtables:<EOL><INDENT>if await t.size() == <NUM_LIT:0>:<EOL><INDENT>return True<EOL><DEDENT><DEDENT>return False<EOL>


fs = (hashtable.itemcounts() for hashtable in self.hashtables)<EOL>return await asyncio.gather(*fs)<EOL>


async def get_subset_counts(self, *keys):

key_set = list(set(keys))<EOL>hashtables = [unordered_storage({'<STR_LIT:type>': '<STR_LIT>'}) for _ in range(self.b)]<EOL>Hss = await self.keys.getmany(*key_set)<EOL>for key, Hs in zip(key_set, Hss):<EOL><INDENT>for H, hashtable in zip(Hs, hashtables):<EOL><INDENT>hashtable.insert(H, key)<EOL><DEDENT><DEDENT>return [hashtable.itemcounts() for hashtable in hashtables]<EOL>


await self.lsh._insert(key, minhash,<EOL>check_duplication=check_duplication, buffer=True)<EOL>


await self.lsh._remove(key, buffer=True)<EOL>


def _compute_nfp_uniform(l, u, cum_counts, sizes):

if l > u:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if l == <NUM_LIT:0>:<EOL><INDENT>n = cum_counts[u]<EOL><DEDENT>else:<EOL><INDENT>n = cum_counts[u]-cum_counts[l-<NUM_LIT:1>]<EOL><DEDENT>return n * float(sizes[u] - sizes[l]) / float(<NUM_LIT:2>*sizes[u])<EOL>

Computes the expected number of false positives caused by using
    u to approximate set sizes in the interval [l, u], assuming uniform
    distribution of set sizes within the interval.

    Args:
        l: the lower bound on set sizes.
        u: the upper bound on set sizes.
        cum_counts: the complete cummulative distribution of set sizes.
        sizes: the complete domain of set sizes.

    Return (float): the expected number of false positives.


nfps = np.zeros((len(sizes), len(sizes)))<EOL>for l in range(len(sizes)):<EOL><INDENT>for u in range(l, len(sizes)):<EOL><INDENT>nfps[l, u] = _compute_nfp_uniform(l, u, cum_counts, sizes)<EOL><DEDENT><DEDENT>return nfps<EOL>

Computes the matrix of expected false positives for all possible
    sub-intervals of the complete domain of set sizes, assuming uniform
    distribution of set_sizes within each sub-intervals.

    Args:
        cum_counts: the complete cummulative distribution of set sizes.
        sizes: the complete domain of set sizes.

    Return (np.array): the 2-D array of expected number of false positives
        for every pair of [l, u] interval, where l is axis-0 and u is
        axis-1.


def _compute_nfp_real(l, u, counts, sizes):

if l > u:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>return np.sum((float(sizes[u])-sizes[l:u+<NUM_LIT:1>])/float(sizes[u])*counts[l:u+<NUM_LIT:1>])<EOL>

Computes the expected number of false positives caused by using
    u to approximate set sizes in the interval [l, u], using the real
    set size distribution.

    Args:
        l: the lower bound on set sizes.
        u: the upper bound on set sizes.
        counts: the complete distribution of set sizes.
        sizes: the complete domain of set sizes.

    Return (float): the expected number of false positives.


nfps = np.zeros((len(sizes), len(sizes)))<EOL>for l in range(len(sizes)):<EOL><INDENT>for u in range(l, len(sizes)):<EOL><INDENT>nfps[l, u] = _compute_nfp_real(l, u, counts, sizes)<EOL><DEDENT><DEDENT>return nfps<EOL>

Computes the matrix of expected false positives for all possible
    sub-intervals of the complete domain of set sizes.

    Args:
        counts: the complete distribution of set sizes.
        sizes: the complete domain of set sizes.

    Return (np.array): the 2-D array of expected number of false positives
        for every pair of [l, u] interval, where l is axis-0 and u is
        axis-1.


def _compute_best_partitions(num_part, sizes, nfps):

if num_part < <NUM_LIT:2>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if num_part > len(sizes):<EOL><INDENT>raise ValueError("<STR_LIT>"<EOL>"<STR_LIT>")<EOL><DEDENT>if num_part == <NUM_LIT:2>:<EOL><INDENT>total_nfps, u = min((nfps[<NUM_LIT:0>, u1]+nfps[u1+<NUM_LIT:1>, len(sizes)-<NUM_LIT:1>], u1)<EOL>for u1 in range(<NUM_LIT:0>, len(sizes)-<NUM_LIT:1>))<EOL>return [(sizes[<NUM_LIT:0>], sizes[u]), (sizes[u+<NUM_LIT:1>], sizes[-<NUM_LIT:1>]),],total_nfps, None<EOL><DEDENT>cost = np.zeros((len(sizes), num_part-<NUM_LIT:2>))<EOL>p2i = lambda p : p - <NUM_LIT:2><EOL>for p in range(<NUM_LIT:2>, num_part):<EOL><INDENT>for u in range(p-<NUM_LIT:1>, len(sizes)):<EOL><INDENT>if p == <NUM_LIT:2>:<EOL><INDENT>cost[u, p2i(p)] = min(nfps[<NUM_LIT:0>, u1]+nfps[u1+<NUM_LIT:1>,u]<EOL>for u1 in range(u))<EOL><DEDENT>else:<EOL><INDENT>cost[u, p2i(p)] = min(cost[u1, p2i(p-<NUM_LIT:1>)] + nfps[u1+<NUM_LIT:1>, u]<EOL>for u1 in range((p-<NUM_LIT:1>)-<NUM_LIT:1>, u))<EOL><DEDENT><DEDENT><DEDENT>p = num_part<EOL>total_nfps, u = min((cost[u1, p2i(p-<NUM_LIT:1>)]+nfps[u1+<NUM_LIT:1>, len(sizes)-<NUM_LIT:1>], u1)<EOL>for u1 in range((p-<NUM_LIT:1>)-<NUM_LIT:1>, len(sizes)-<NUM_LIT:1>))<EOL>partitions = [(sizes[u+<NUM_LIT:1>], sizes[-<NUM_LIT:1>]),]<EOL>p -= <NUM_LIT:1><EOL>while p > <NUM_LIT:1>:<EOL><INDENT>_, u1_best = min((cost[u1, p2i(p)]+nfps[u1+<NUM_LIT:1>, u], u1)<EOL>for u1 in range((p-<NUM_LIT:1>)-<NUM_LIT:1>, u))<EOL>partitions.insert(<NUM_LIT:0>, (sizes[u1_best+<NUM_LIT:1>], sizes[u]))<EOL>u = u1_best<EOL>p -= <NUM_LIT:1><EOL><DEDENT>partitions.insert(<NUM_LIT:0>, (sizes[<NUM_LIT:0>], sizes[u]))<EOL>return [partitions, total_nfps, cost]<EOL>

Computes the optimal partitions given the size distributions
    and computed number of expected false positives for all sub-intervals.

    Args:
        num_part (int): The number of partitions to create.
        sizes (numpy.array): The complete domain of set sizes in sorted order.
        nfps (numpy.array): The computed number of expected false positives
            for all sub-intervals; axis-0 is for the indexes of lower bounds and
            axis-1 is for the indexes of upper bounds.

    Returns:
        partitions (list): list of lower and upper bounds of set sizes for
            all partitions.
        total_nfps (float): total number of expected false positives from all
            partitions.
        cost (numpy.array): a N x p-1 matrix of the computed optimal NFPs for
            all sub-problems given upper bound set size and number of partitions.


def optimal_partitions(sizes, counts, num_part):

if num_part < <NUM_LIT:2>:<EOL><INDENT>return [(sizes[<NUM_LIT:0>], sizes[-<NUM_LIT:1>])]<EOL><DEDENT>if num_part >= len(sizes):<EOL><INDENT>partitions = [(x, x) for x in sizes]<EOL>return partitions<EOL><DEDENT>nfps = _compute_nfps_real(counts, sizes)<EOL>partitions, _, _ = _compute_best_partitions(num_part, sizes, nfps)<EOL>return partitions<EOL>

Compute the optimal partitions given a distribution of set sizes.

    Args:
        sizes (numpy.array): The complete domain of set sizes in ascending
            order.
        counts (numpy.array): The frequencies of all set sizes in the same
            order as `sizes`.
        num_part (int): The number of partitions to create.

    Returns:
        list: A list of partitions in the form of `(lower, upper)` tuples,
            where `lower` and `upper` are lower and upper bound (inclusive)
            set sizes of each partition.


tp = config['<STR_LIT:type>']<EOL>if tp == '<STR_LIT>':<EOL><INDENT>return DictListStorage(config)<EOL><DEDENT>if tp == '<STR_LIT>':<EOL><INDENT>return RedisListStorage(config, name=name)<EOL><DEDENT>

Return ordered storage system based on the specified config.

    The canonical example of such a storage container is
    ``defaultdict(list)``. Thus, the return value of this method contains
    keys and values. The values are ordered lists with the last added
    item at the end.

    Args:
        config (dict): Defines the configurations for the storage.
            For in-memory storage, the config ``{'type': 'dict'}`` will
            suffice. For Redis storage, the type should be ``'redis'`` and
            the configurations for the Redis database should be supplied
            under the key ``'redis'``. These parameters should be in a form
            suitable for `redis.Redis`. The parameters may alternatively
            contain references to environment variables, in which case
            literal configuration values should be replaced by dicts of
            the form::

                {'env': 'REDIS_HOSTNAME',
                 'default': 'localhost'}

            For a full example, see :ref:`minhash_lsh_at_scale`

        name (bytes, optional): A reference name for this storage container.
            For dict-type containers, this is ignored. For Redis containers,
            this name is used to prefix keys pertaining to this storage
            container within the database.


tp = config['<STR_LIT:type>']<EOL>if tp == '<STR_LIT>':<EOL><INDENT>return DictSetStorage(config)<EOL><DEDENT>if tp == '<STR_LIT>':<EOL><INDENT>return RedisSetStorage(config, name=name)<EOL><DEDENT>

Return an unordered storage system based on the specified config.

    The canonical example of such a storage container is
    ``defaultdict(set)``. Thus, the return value of this method contains
    keys and values. The values are unordered sets.

    Args:
        config (dict): Defines the configurations for the storage.
            For in-memory storage, the config ``{'type': 'dict'}`` will
            suffice. For Redis storage, the type should be ``'redis'`` and
            the configurations for the Redis database should be supplied
            under the key ``'redis'``. These parameters should be in a form
            suitable for `redis.Redis`. The parameters may alternatively
            contain references to environment variables, in which case
            literal configuration values should be replaced by dicts of
            the form::

                {'env': 'REDIS_HOSTNAME',
                 'default': 'localhost'}

            For a full example, see :ref:`minhash_lsh_at_scale`

        name (bytes, optional): A reference name for this storage container.
            For dict-type containers, this is ignored. For Redis containers,
            this name is used to prefix keys pertaining to this storage
            container within the database.


return []<EOL>


pass<EOL>

Get list of values associated with a key

        Returns empty list ([]) if `key` is not found


@abstractmethod<EOL><INDENT>def insert(self, key, *vals, **kwargs):<DEDENT>

pass<EOL>


pass<EOL>


@abstractmethod<EOL><INDENT>def remove_val(self, key, val):<DEDENT>

pass<EOL>

Remove `val` from list of values under `key`


pass<EOL>

Return size of storage with respect to number of keys


pass<EOL>

Returns the number of items stored under each key


pass<EOL>

Determines whether the key is in the storage or not


return {k: len(v) for k, v in self._dict.items()}<EOL>

Returns a dict where the keys are the keys of the container.
        The values are the *lengths* of the value sequences stored
        in this container.


def _false_positive_probability(threshold, b, r, xq):

_probability = lambda t : <NUM_LIT:1> - (<NUM_LIT:1> - (t/(<NUM_LIT:1> + xq - t))**float(r))**float(b)<EOL>if xq >= threshold:<EOL><INDENT>a, err = integrate(_probability, <NUM_LIT:0.0>, threshold)<EOL>return a<EOL><DEDENT>a, err = integrate(_probability, <NUM_LIT:0.0>, xq)<EOL>return a<EOL>

Compute the false positive probability given the containment threshold.
xq is the ratio of x/q.


def _false_negative_probability(threshold, b, r, xq):

_probability = lambda t : <NUM_LIT:1> - (<NUM_LIT:1> - (<NUM_LIT:1> - (t/(<NUM_LIT:1> + xq - t))**float(r))**float(b))<EOL>if xq >= <NUM_LIT:1.0>:<EOL><INDENT>a, err = integrate(_probability, threshold, <NUM_LIT:1.0>)<EOL>return a<EOL><DEDENT>if xq >= threshold:<EOL><INDENT>a, err = integrate(_probability, threshold, xq)<EOL>return a<EOL><DEDENT>return <NUM_LIT:0.0><EOL>

Compute the false negative probability given the containment threshold


def _optimal_param(threshold, num_perm, max_r, xq, false_positive_weight,<EOL>false_negative_weight):

min_error = float("<STR_LIT>")<EOL>opt = (<NUM_LIT:0>, <NUM_LIT:0>)<EOL>for b in range(<NUM_LIT:1>, num_perm+<NUM_LIT:1>):<EOL><INDENT>for r in range(<NUM_LIT:1>, max_r+<NUM_LIT:1>):<EOL><INDENT>if b*r > num_perm:<EOL><INDENT>continue<EOL><DEDENT>fp = _false_positive_probability(threshold, b, r, xq)<EOL>fn = _false_negative_probability(threshold, b, r, xq)<EOL>error = fp*false_positive_weight + fn*false_negative_weight<EOL>if error < min_error:<EOL><INDENT>min_error = error<EOL>opt = (b, r)<EOL><DEDENT><DEDENT><DEDENT>return opt<EOL>

Compute the optimal parameters that minimizes the weighted sum
of probabilities of false positive and false negative.
xq is the ratio of x/q.


if not self.is_empty():<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if not isinstance(entries, list):<EOL><INDENT>queue = deque([])<EOL>for key, minhash, size in entries:<EOL><INDENT>if size <= <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>queue.append((key, minhash, size))<EOL><DEDENT>entries = list(queue)<EOL><DEDENT>if len(entries) == <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>sizes, counts = np.array(sorted(<EOL>Counter(e[<NUM_LIT:2>] for e in entries).most_common())).T<EOL>partitions = optimal_partitions(sizes, counts, len(self.indexes))<EOL>for i, (lower, upper) in enumerate(partitions):<EOL><INDENT>self.lowers[i], self.uppers[i] = lower, upper<EOL><DEDENT>entries.sort(key=lambda e : e[<NUM_LIT:2>])<EOL>curr_part = <NUM_LIT:0><EOL>for key, minhash, size in entries:<EOL><INDENT>if size > self.uppers[curr_part]:<EOL><INDENT>curr_part += <NUM_LIT:1><EOL><DEDENT>for r in self.indexes[curr_part]:<EOL><INDENT>self.indexes[curr_part][r].insert(key, minhash)<EOL><DEDENT><DEDENT>

Index all sets given their keys, MinHashes, and sizes.
It can be called only once after the index is created.

Args:
    entries (`iterable` of `tuple`): An iterable of tuples, each must be
        in the form of `(key, minhash, size)`, where `key` is the unique
        identifier of a set, `minhash` is the MinHash of the set,
        and `size` is the size or number of unique items in the set.

Note:
    `size` must be positive.


for i, index in enumerate(self.indexes):<EOL><INDENT>u = self.uppers[i]<EOL>if u is None:<EOL><INDENT>continue<EOL><DEDENT>b, r = self._get_optimal_param(u, size)<EOL>for key in index[r]._query_b(minhash, b):<EOL><INDENT>yield key<EOL><DEDENT><DEDENT>

Giving the MinHash and size of the query set, retrieve
keys that references sets with containment with respect to
the query set greater than the threshold.

Args:
    minhash (datasketch.MinHash): The MinHash of the query set.
    size (int): The size (number of unique items) of the query set.

Returns:
    `iterator` of keys.


return any(any(key in index[r] for r in index)<EOL>for index in self.indexes)<EOL>

Args:
    key (hashable): The unique identifier of a set.

Returns:
    bool: True only if the key exists in the index.


return all(all(index[r].is_empty() for r in index)<EOL>for index in self.indexes)<EOL>

Returns:
    bool: Check if the index is empty.


hv = self.hashfunc(b)<EOL>a, b = self.permutations<EOL>phv = np.bitwise_and((a * hv + b) % _mersenne_prime, np.uint64(_max_hash))<EOL>self.hashvalues = np.minimum(phv, self.hashvalues)<EOL>

Update this MinHash with a new value.
        The value will be hashed using the hash function specified by
        the `hashfunc` argument in the constructor.

        Args:
            b: The value to be hashed using the hash function specified.

        Example:
            To update with a new string value (using the default SHA1 hash
            function, which requires bytes as input):

            .. code-block:: python

                minhash = Minhash()
                minhash.update("new value".encode('utf-8'))

            We can also use a different hash function, for example, `pyfarmhash`:

            .. code-block:: python

                import farmhash
                def _hash_32(b):
                    return farmhash.hash32(b)
                minhash = MinHash(hashfunc=_hash_32)
                minhash.update("new value")


if other.seed != self.seed:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if len(self) != len(other):<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>return np.float(np.count_nonzero(self.hashvalues==other.hashvalues)) /np.float(len(self))<EOL>

Estimate the `Jaccard similarity`_ (resemblance) between the sets
        represented by this MinHash and the other.

        Args:
            other (datasketch.MinHash): The other MinHash.

        Returns:
            float: The Jaccard similarity, which is between 0.0 and 1.0.


k = len(self)<EOL>return np.float(k) / np.sum(self.hashvalues / np.float(_max_hash)) - <NUM_LIT:1.0><EOL>

Estimate the cardinality count based on the technique described in
        `this paper <http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=365694>`_.

        Returns:
            int: The estimated cardinality of the set represented by this MinHash.


if other.seed != self.seed:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if len(self) != len(other):<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>self.hashvalues = np.minimum(other.hashvalues, self.hashvalues)<EOL>

Merge the other MinHash with this one, making this one the union
        of both.

        Args:
            other (datasketch.MinHash): The other MinHash.


return copy.copy(self.hashvalues)<EOL>

Export the hash values, which is the internal state of the
        MinHash.

        Returns:
            numpy.array: The hash values which is a Numpy array.


if np.any(self.hashvalues != _max_hash):<EOL><INDENT>return False<EOL><DEDENT>return True<EOL>

Returns:
    bool: If the current MinHash is empty - at the state of just
        initialized.


self.hashvalues = self._init_hashvalues(len(self))<EOL>

Clear the current state of the MinHash.
All hash values are reset.


return MinHash(seed=self.seed, hashfunc=self.hashfunc,<EOL>hashvalues=self.digest(),<EOL>permutations=self.permutations)<EOL>

:returns: datasketch.MinHash -- A copy of this MinHash by exporting its state.


return len(self.hashvalues)<EOL>

:returns: int -- The number of hash values.


return type(self) is type(other) andself.seed == other.seed andnp.array_equal(self.hashvalues, other.hashvalues)<EOL>

:returns: bool -- If their seeds and hash values are both equal then two are equivalent.


if len(mhs) < <NUM_LIT:2>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>num_perm = len(mhs[<NUM_LIT:0>])<EOL>seed = mhs[<NUM_LIT:0>].seed<EOL>if any((seed != m.seed or num_perm != len(m)) for m in mhs):<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>hashvalues = np.minimum.reduce([m.hashvalues for m in mhs])<EOL>permutations = mhs[<NUM_LIT:0>].permutations<EOL>return cls(num_perm=num_perm, seed=seed, hashvalues=hashvalues,<EOL>permutations=permutations)<EOL>

Create a MinHash which is the union of the MinHash objects passed as arguments.

        Args:
            *mhs: The MinHash objects to be united. The argument list length is variable,
                but must be at least 2.

        Returns:
            datasketch.MinHash: A new union MinHash.


def _initialize_slots(self, seed, hashvalues):

self.seed = seed<EOL>self.hashvalues = self._parse_hashvalues(hashvalues)<EOL>

Initialize the slots of the LeanMinHash.

        Args:
            seed (int): The random seed controls the set of random
                permutation functions generated for this LeanMinHash.
            hashvalues: The hash values is the internal state of the LeanMinHash.


raise TypeError("<STR_LIT>")<EOL>

This method is not available on a LeanMinHash.
        Calling it raises a TypeError.


<EOL>seed_size = struct.calcsize(byteorder+'<STR_LIT:q>')<EOL>length_size = struct.calcsize(byteorder+'<STR_LIT:i>')<EOL>hashvalue_size = struct.calcsize(byteorder+'<STR_LIT:I>')<EOL>return seed_size + length_size + len(self) * hashvalue_size<EOL>

Compute the byte size after serialization.

        Args:
            byteorder (str, optional): This is byte order of the serialized data. Use one
                of the `byte order characters
                <https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment>`_:
                ``@``, ``=``, ``<``, ``>``, and ``!``.
                Default is ``@`` -- the native order.

        Returns:
            int: Size in number of bytes after serialization.


def serialize(self, buf, byteorder='<STR_LIT:@>'):

if len(buf) < self.bytesize():<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>fmt = "<STR_LIT>" % (byteorder, len(self))<EOL>struct.pack_into(fmt, buf, <NUM_LIT:0>,<EOL>self.seed, len(self), *self.hashvalues)<EOL>

Serialize this lean MinHash and store the result in an allocated buffer.

Args:
    buf (buffer): `buf` must implement the `buffer`_ interface.
        One such example is the built-in `bytearray`_ class.
    byteorder (str, optional): This is byte order of the serialized data. Use one
        of the `byte order characters
        <https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment>`_:
        ``@``, ``=``, ``<``, ``>``, and ``!``.
        Default is ``@`` -- the native order.

This is preferred over using `pickle`_ if the serialized lean MinHash needs
to be used by another program in a different programming language.

The serialization schema:
    1. The first 8 bytes is the seed integer
    2. The next 4 bytes is the number of hash values
    3. The rest is the serialized hash values, each uses 4 bytes

Example:
    To serialize a single lean MinHash into a `bytearray`_ buffer.

    .. code-block:: python

        buf = bytearray(lean_minhash.bytesize())
        lean_minhash.serialize(buf)

    To serialize multiple lean MinHash into a `bytearray`_ buffer.

    .. code-block:: python

        # assuming lean_minhashs is a list of LeanMinHash with the same size
        size = lean_minhashs[0].bytesize()
        buf = bytearray(size*len(lean_minhashs))
        for i, lean_minhash in enumerate(lean_minhashs):
            lean_minhash.serialize(buf[i*size:])

.. _`buffer`: https://docs.python.org/3/c-api/buffer.html
.. _`bytearray`: https://docs.python.org/3.6/library/functions.html#bytearray
.. _`byteorder`: https://docs.python.org/3/library/struct.html


@classmethod<EOL><INDENT>def deserialize(cls, buf, byteorder='<STR_LIT:@>'):<DEDENT>

fmt_seed_size = "<STR_LIT>" % byteorder<EOL>fmt_hash = byteorder + "<STR_LIT>"<EOL>try:<EOL><INDENT>seed, num_perm = struct.unpack_from(fmt_seed_size, buf, <NUM_LIT:0>)<EOL><DEDENT>except TypeError:<EOL><INDENT>seed, num_perm = struct.unpack_from(fmt_seed_size, buffer(buf), <NUM_LIT:0>)<EOL><DEDENT>offset = struct.calcsize(fmt_seed_size)<EOL>try:<EOL><INDENT>hashvalues = struct.unpack_from(fmt_hash % num_perm, buf, offset)<EOL><DEDENT>except TypeError:<EOL><INDENT>hashvalues = struct.unpack_from(fmt_hash % num_perm, buffer(buf), offset)<EOL><DEDENT>lmh = object.__new__(LeanMinHash)<EOL>lmh._initialize_slots(seed, hashvalues)<EOL>return lmh<EOL>

Deserialize a lean MinHash from a buffer.

Args:
    buf (buffer): `buf` must implement the `buffer`_ interface.
        One such example is the built-in `bytearray`_ class.
    byteorder (str. optional): This is byte order of the serialized data. Use one
        of the `byte order characters
        <https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment>`_:
        ``@``, ``=``, ``<``, ``>``, and ``!``.
        Default is ``@`` -- the native order.

Return:
    datasketch.LeanMinHash: The deserialized lean MinHash

Example:
    To deserialize a lean MinHash from a buffer.

    .. code-block:: python

        lean_minhash = LeanMinHash.deserialize(buf)


def __init__(self, minhash, b=<NUM_LIT:1>, r=<NUM_LIT:0.0>):

b = int(b)<EOL>r = float(r)<EOL>if b > <NUM_LIT:32> or b < <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if r > <NUM_LIT:1.0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>bmask = (<NUM_LIT:1> << b) - <NUM_LIT:1><EOL>self.hashvalues = np.bitwise_and(minhash.hashvalues, bmask).astype(np.uint32)<EOL>self.seed = minhash.seed<EOL>self.b = b<EOL>self.r = r<EOL>

Initialize a b-bit MinHash given an existing full MinHash
object and parameter b - the number of bits to store for
each minimum hashed values in the MinHash object.


return type(self) is type(other) andself.seed == other.seed andself.b == other.b andself.r == other.r andnp.array_equal(self.hashvalues, other.hashvalues)<EOL>

Check for full equality of two b-bit MinHash objects.


if self.b != other.b:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if self.seed != other.seed:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>intersection = np.count_nonzero(self.hashvalues==other.hashvalues)<EOL>raw_est = float(intersection) / float(self.hashvalues.size)<EOL>a1 = self._calc_a(self.r, self.b)<EOL>a2 = self._calc_a(other.r, other.b)<EOL>c1, c2 = self._calc_c(a1, a2, self.r, other.r)<EOL>return (raw_est - c1) / (<NUM_LIT:1> - c2)<EOL>

Estimate the Jaccard similarity (resemblance) between this b-bit
MinHash and the other.


return self._bytesize()[-<NUM_LIT:1>]<EOL>

Get the serialized size of this b-bit MinHash in number of bytes.


slot_size, n, num_blocks, total = self._bytesize()<EOL>buffer = bytearray(total)<EOL>blocks = np.zeros((num_blocks,), dtype=np.uint64)<EOL>for i in range(num_blocks):<EOL><INDENT>start = i * n<EOL>hvs = self.hashvalues[start:start+n]<EOL>for j, hv in enumerate(hvs):<EOL><INDENT>blocks[i] |= np.uint64(hv << (n - <NUM_LIT:1> - j) * slot_size)<EOL><DEDENT><DEDENT>fmt = self._serial_fmt_params +"<STR_LIT>" % (num_blocks, self._serial_fmt_block)<EOL>struct.pack_into(fmt, buffer, <NUM_LIT:0>, self.seed, self.b, self.r,self.hashvalues.size, *blocks)<EOL>return buffer<EOL>

This function is called when pickling the b-bit MinHash object.
Returns a bytearray which will then be pickled.


try:<EOL><INDENT>self.seed, self.b, self.r, num_perm =struct.unpack_from(self._serial_fmt_params, buf, <NUM_LIT:0>)<EOL><DEDENT>except TypeError:<EOL><INDENT>self.seed, self.b, self.r, num_perm =struct.unpack_from(self._serial_fmt_params, buffer(buf), <NUM_LIT:0>)<EOL><DEDENT>offset = struct.calcsize(self._serial_fmt_params)<EOL>self.hashvalues = np.zeros((num_perm,), dtype=np.uint32)<EOL>slot_size, n, num_blocks, total = self._bytesize()<EOL>fmt = "<STR_LIT>" % (num_blocks, self._serial_fmt_block)<EOL>try:<EOL><INDENT>blocks = struct.unpack_from(fmt, buf, offset)<EOL><DEDENT>except TypeError:<EOL><INDENT>blocks = struct.unpack_from(fmt, buffer(buf), offset)<EOL><DEDENT>mask = (<NUM_LIT:1> << slot_size) - <NUM_LIT:1><EOL>for i in range(num_blocks):<EOL><INDENT>start = i * n<EOL>for j, _ in enumerate(self.hashvalues[start:start+n]):<EOL><INDENT>hv = (blocks[i] >> (n - <NUM_LIT:1> - j) * slot_size) & mask<EOL>self.hashvalues[start+j] = np.uint32(hv)<EOL><DEDENT><DEDENT>

This function is called when unpickling the b-bit MinHash object.
Initialize the object with data in the buffer.


if r == <NUM_LIT:0.0>:<EOL><INDENT>return <NUM_LIT:1.0> / (<NUM_LIT:1> << b)<EOL><DEDENT>return r * (<NUM_LIT:1> - r) ** (<NUM_LIT:2> ** b - <NUM_LIT:1>) / (<NUM_LIT:1> - (<NUM_LIT:1> - r) ** (<NUM_LIT:2> * b))<EOL>


if r1 == <NUM_LIT:0.0> and r2 == <NUM_LIT:0.0>:<EOL><INDENT>return a1, a2<EOL><DEDENT>div = <NUM_LIT:1> / (r1 + r2)<EOL>c1 = (a1 * r2 + a2 * r1) * div<EOL>c2 = (a1 * r1 + a2 * r2) * div<EOL>return c1, c2<EOL>


if len(minhash) < self.k*self.l:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if key in self.keys:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>self.keys[key] = [self._H(minhash.hashvalues[start:end])<EOL>for start, end in self.hashranges]<EOL>for H, hashtable in zip(self.keys[key], self.hashtables):<EOL><INDENT>hashtable[H].append(key)<EOL><DEDENT>

Add a unique key, together
with a MinHash (or weighted MinHash) of the set referenced by the key.

Note:
    The key won't be searchbale until the
    :func:`datasketch.MinHashLSHForest.index` method is called.

Args:
    key (hashable): The unique identifier of the set.
    minhash (datasketch.MinHash): The MinHash of the set.


for i, hashtable in enumerate(self.hashtables):<EOL><INDENT>self.sorted_hashtables[i] = [H for H in hashtable.keys()]<EOL>self.sorted_hashtables[i].sort()<EOL><DEDENT>

Index all the keys added so far and make them searchable.


if k <= <NUM_LIT:0>:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>if len(minhash) < self.k*self.l:<EOL><INDENT>raise ValueError("<STR_LIT>")<EOL><DEDENT>results = set()<EOL>r = self.k<EOL>while r > <NUM_LIT:0>:<EOL><INDENT>for key in self._query(minhash, r, self.l):<EOL><INDENT>results.add(key)<EOL>if len(results) >= k:<EOL><INDENT>return list(results)<EOL><DEDENT><DEDENT>r -= <NUM_LIT:1><EOL><DEDENT>return list(results)<EOL>

Return the approximate top-k keys that have the highest
Jaccard similarities to the query set.

Args:
    minhash (datasketch.MinHash): The MinHash of the query set.
    k (int): The maximum number of keys to return.

Returns:
    `list` of at most k keys.


i, j = <NUM_LIT:0>, n<EOL>while i < j:<EOL><INDENT>h = int(i + (j - i) / <NUM_LIT:2>)<EOL>if not func(h):<EOL><INDENT>i = h + <NUM_LIT:1><EOL><DEDENT>else:<EOL><INDENT>j = h<EOL><DEDENT><DEDENT>return i<EOL>


return any(len(t) == <NUM_LIT:0> for t in self.sorted_hashtables)<EOL>

Check whether there is any searchable keys in the index.
Note that keys won't be searchable until `index` is called.

Returns:
    bool: True if there is no searchable key in the index.


return key in self.keys<EOL>

Returns:
    bool: True only if the key has been added to the index.


m = len(pvalues)<EOL>pi0 = (pvalues > lamb).sum() / ((<NUM_LIT:1> - lamb)*m)<EOL>pFDR = np.ones(m)<EOL>print("<STR_LIT>")<EOL>for i in range(m):<EOL><INDENT>y = pvalues[i]<EOL>Pr = max(<NUM_LIT:1>, m - i) / float(m)<EOL>pFDR[i] = (pi0 * y) / (Pr * (<NUM_LIT:1> - math.pow(<NUM_LIT:1>-y, m)))<EOL>print(i, pFDR[i], y, Pr, <NUM_LIT:1.0> - math.pow(<NUM_LIT:1>-y, m))<EOL><DEDENT>num_null = pi0*m<EOL>num_alt = m - num_null<EOL>num_negs = np.array(range(m))<EOL>num_pos = m - num_negs<EOL>pp = num_pos / float(m)<EOL>qvalues = np.ones(m)<EOL>qvalues[<NUM_LIT:0>] = pFDR[<NUM_LIT:0>]<EOL>for i in range(m-<NUM_LIT:1>):<EOL><INDENT>qvalues[i+<NUM_LIT:1>] = min(qvalues[i], pFDR[i+<NUM_LIT:1>])<EOL><DEDENT>sens = ((<NUM_LIT:1.0> - qvalues) * num_pos) / num_alt<EOL>sens[sens > <NUM_LIT:1.0>] = <NUM_LIT:1.0><EOL>df = pd.DataFrame(dict(<EOL>pvalue=pvalues,<EOL>qvalue=qvalues,<EOL>FDR=pFDR,<EOL>percentile_positive=pp,<EOL>sens=sens<EOL>))<EOL>df["<STR_LIT>"] = df.sens[::-<NUM_LIT:1>].cummax()[::-<NUM_LIT:1>]<EOL>return df, num_null, m<EOL>

meaning pvalues presorted i descending order


(inst, method_name, args) = arg<EOL>return getattr(inst, method_name)(*args)<EOL>

You can not call methods with multiprocessing, but free functions,
        If you want to call  inst.method(arg0, arg1),

            unwrap_self_for_multiprocessing(inst, "method", (arg0, arg1))

        does the trick.


data = vars(self)<EOL>data["<STR_LIT>"] = self.minimal_error_stat()<EOL>return data<EOL>


PyProphet: Semi-supervised learning and scoring of OpenSWATH results.

Visit http://openswath.org for usage instructions and help.


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=True, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=True, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=True, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>def ipf(infile, outfile, ipf_ms1_scoring, ipf_ms2_scoring, ipf_h0, ipf_grouped_fdr, ipf_max_precursor_pep, ipf_max_peakgroup_pep, ipf_max_precursor_peakgroup_pep, ipf_max_transition_pep):

if outfile is None:<EOL><INDENT>outfile = infile<EOL><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>infer_peptidoforms(infile, outfile, ipf_ms1_scoring, ipf_ms2_scoring, ipf_h0, ipf_grouped_fdr, ipf_max_precursor_pep, ipf_max_peakgroup_pep, ipf_max_precursor_peakgroup_pep, ipf_max_transition_pep)<EOL>

Infer peptidoforms after scoring of MS1, MS2 and transition-level data.


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>', '<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=[<NUM_LIT:0.1>,<NUM_LIT:0.5>,<NUM_LIT>], show_default=True, type=(float, float, float), help='<STR_LIT>', callback=transform_pi0_lambda)<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT:3>, show_default=True, type=int, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', show_default=True, default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', show_default=True, default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=np.power(<NUM_LIT>,-<NUM_LIT:8>), show_default=True, type=float, help='<STR_LIT>')<EOL>def peptide(infile, outfile, context, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0, lfdr_truncate, lfdr_monotone, lfdr_transformation, lfdr_adj, lfdr_eps):

if outfile is None:<EOL><INDENT>outfile = infile<EOL><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>infer_peptides(infile, outfile, context, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0, lfdr_truncate, lfdr_monotone, lfdr_transformation, lfdr_adj, lfdr_eps)<EOL>

Infer peptides and conduct error-rate estimation in different contexts.


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>', '<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=[<NUM_LIT:0.1>,<NUM_LIT:0.5>,<NUM_LIT>], show_default=True, type=(float, float, float), help='<STR_LIT>', callback=transform_pi0_lambda)<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT:3>, show_default=True, type=int, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', show_default=True, default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', show_default=True, default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=np.power(<NUM_LIT>,-<NUM_LIT:8>), show_default=True, type=float, help='<STR_LIT>')<EOL>def protein(infile, outfile, context, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0, lfdr_truncate, lfdr_monotone, lfdr_transformation, lfdr_adj, lfdr_eps):

if outfile is None:<EOL><INDENT>outfile = infile<EOL><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>infer_proteins(infile, outfile, context, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0, lfdr_truncate, lfdr_monotone, lfdr_transformation, lfdr_adj, lfdr_eps)<EOL>

Infer proteins and conduct error-rate estimation in different contexts.


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>','<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>def reduce(infile, outfile):

if outfile is None:<EOL><INDENT>outfile = infile<EOL><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>reduce_osw(infile, outfile)<EOL>

Reduce scored PyProphet file to minimum for global scoring


@cli.command()<EOL>@click.argument('<STR_LIT>', nargs=-<NUM_LIT:1>, type=click.Path(exists=True))<EOL>@click.option('<STR_LIT>','<STR_LIT>', required=True, type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=False, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>','<STR_LIT>', required=True, type=click.Path(exists=False), help='<STR_LIT>')<EOL>def merge(infiles, outfile, same_run, templatefile):

if len(infiles) < <NUM_LIT:1>:<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>merge_osw(infiles, outfile, templatefile, same_run)<EOL>

Merge multiple OSW files and (for large experiments, it is recommended to subsample first).


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>','<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>def backpropagate(infile, outfile, apply_scores):

if outfile is None:<EOL><INDENT>outfile = infile<EOL><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>backpropagate_oswr(infile, outfile, apply_scores)<EOL>

Backpropagate multi-run peptide and protein scores to single files


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>', '<STR_LIT>','<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=True, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>','<STR_LIT>','<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=True, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=True, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>def export(infile, outfile, format, outcsv, transition_quantification, max_transition_pep, ipf, ipf_max_peptidoform_pep, max_rs_peakgroup_qvalue, peptide, max_global_peptide_qvalue, protein, max_global_protein_qvalue):

if format == "<STR_LIT>":<EOL><INDENT>export_score_plots(infile)<EOL><DEDENT>else:<EOL><INDENT>if outfile is None:<EOL><INDENT>if outcsv:<EOL><INDENT>outfile = infile.split("<STR_LIT>")[<NUM_LIT:0>] + "<STR_LIT>"<EOL><DEDENT>else:<EOL><INDENT>outfile = infile.split("<STR_LIT>")[<NUM_LIT:0>] + "<STR_LIT>"<EOL><DEDENT><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>export_tsv(infile, outfile, format, outcsv, transition_quantification, max_transition_pep, ipf, ipf_max_peptidoform_pep, max_rs_peakgroup_qvalue, peptide, max_global_peptide_qvalue, protein, max_global_protein_qvalue)<EOL><DEDENT>


@cli.command()<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', type=click.Path(exists=False), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default='<STR_LIT>', show_default=True, type=click.Choice(['<STR_LIT>', '<STR_LIT>','<STR_LIT>']), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', '<STR_LIT>', default=False, show_default=True, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>def export_compound(infile, outfile, format, outcsv, max_rs_peakgroup_qvalue):

if format == "<STR_LIT>":<EOL><INDENT>export_score_plots(infile)<EOL><DEDENT>else:<EOL><INDENT>if outfile is None:<EOL><INDENT>if outcsv:<EOL><INDENT>outfile = infile.split("<STR_LIT>")[<NUM_LIT:0>] + "<STR_LIT>"<EOL><DEDENT>else:<EOL><INDENT>outfile = infile.split("<STR_LIT>")[<NUM_LIT:0>] + "<STR_LIT>"<EOL><DEDENT><DEDENT>else:<EOL><INDENT>outfile = outfile<EOL><DEDENT>export_compound_tsv(infile, outfile, format, outcsv, max_rs_peakgroup_qvalue)<EOL><DEDENT>


@cli.command()<EOL>@click.argument('<STR_LIT>', nargs=-<NUM_LIT:1>, type=click.Path(exists=True))<EOL>@click.option('<STR_LIT>', '<STR_LIT>', required=True, type=click.Path(exists=True), help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>@click.option('<STR_LIT>', default=<NUM_LIT>, show_default=True, type=float, help='<STR_LIT>')<EOL>def filter(sqmassfiles, infile, max_precursor_pep, max_peakgroup_pep, max_transition_pep):

filter_sqmass(sqmassfiles, infile, max_precursor_pep, max_peakgroup_pep, max_transition_pep)<EOL>


if isinstance(values, (list, tuple)):<EOL><INDENT>values = np.array(values, dtype=np.float32)<EOL><DEDENT>elif isinstance(values, pd.Series):<EOL><INDENT>values = values.values<EOL><DEDENT>values = values.flatten()<EOL>assert values.ndim == <NUM_LIT:1>, "<STR_LIT>"<EOL>if as_type is not None:<EOL><INDENT>return values.astype(as_type)<EOL><DEDENT>return values<EOL>

Converts list or flattens n-dim array to 1-dim array if possible


ix = find_nearest_matches(np.float32(err_df.cutoff.values), np.float32(scores))<EOL>return err_df.pvalue.iloc[ix].values, err_df.svalue.iloc[ix].values, err_df.pep.iloc[ix].values, err_df.qvalue.iloc[ix].values<EOL>

Find matching q-value for each score in 'scores


tg_ids = experiment.df.tg_num_id.values<EOL>pp_values = <NUM_LIT:1>-experiment.df["<STR_LIT>"].values<EOL>current_tg_id = tg_ids[<NUM_LIT:0>]<EOL>scores = []<EOL>final_result = []<EOL>final_result_h0 = []<EOL>for i in range(tg_ids.shape[<NUM_LIT:0>]):<EOL><INDENT>id_ = tg_ids[i]<EOL>if id_ != current_tg_id:<EOL><INDENT>prior_pg_true = (<NUM_LIT:1.0> - prior_chrom_null) / len(scores)<EOL>rr = single_chromatogram_hypothesis_fast(<EOL>np.array(scores), prior_chrom_null, prior_pg_true)<EOL>final_result.extend(rr[<NUM_LIT:1>:])<EOL>final_result_h0.extend(rr[<NUM_LIT:0>] for i in range(len(scores)))<EOL>scores = []<EOL>current_tg_id = id_<EOL><DEDENT>scores.append(<NUM_LIT:1.0> - pp_values[i])<EOL><DEDENT>prior_pg_true = (<NUM_LIT:1.0> - prior_chrom_null) / len(scores)<EOL>rr = single_chromatogram_hypothesis_fast(np.array(scores), prior_chrom_null, prior_pg_true)<EOL>final_result.extend(rr[<NUM_LIT:1>:])<EOL>final_result_h0.extend([rr[<NUM_LIT:0>]] * len(scores))<EOL>return final_result, final_result_h0<EOL>

Compute posterior probabilities for each chromatogram

    For each chromatogram (each group_id / peptide precursor), all hypothesis of all peaks
    being correct (and all others false) as well as the h0 (all peaks are
    false) are computed.

    The prior probability that the  are given in the function

    This assumes that the input data is sorted by tg_num_id

        Args:
            experiment(:class:`data_handling.Multipeptide`): the data of one experiment
            prior_chrom_null(float): the prior probability that any precursor
                is absent (all peaks are false)

        Returns:
            tuple(hypothesis, h0): two vectors that contain for each entry in
            the input dataframe the probabilities for the hypothesis that the
            peak is correct and the probability for the h0


mu, sigma = mean_and_std_dev(stat0)<EOL>stat = to_one_dim_array(stat, np.float64)<EOL>args = (stat - mu) / sigma<EOL>return <NUM_LIT:1>-(<NUM_LIT:0.5> * (<NUM_LIT:1.0> + scipy.special.erf(args / np.sqrt(<NUM_LIT>))))<EOL>

[P(X>pi, mu, sigma) for pi in pvalues] for normal distributed stat with
    expectation value mu and std deviation sigma


assert len(stat0) > <NUM_LIT:0><EOL>assert len(stat) > <NUM_LIT:0><EOL>stat = np.array(stat)<EOL>stat0 = np.array(stat0)<EOL>m = len(stat)<EOL>m0 = len(stat0)<EOL>statc = np.concatenate((stat, stat0))<EOL>v = np.array([True] * m + [False] * m0)<EOL>perm = np.argsort(-statc, kind="<STR_LIT>")  <EOL>v = v[perm]<EOL>u = np.where(v)[<NUM_LIT:0>]<EOL>p = (u - np.arange(m)) / float(m0)<EOL>ranks = np.floor(scipy.stats.rankdata(-stat)).astype(int) - <NUM_LIT:1><EOL>p = p[ranks]<EOL>p[p <= <NUM_LIT:1.0> / m0] = <NUM_LIT:1.0> / m0<EOL>return p<EOL>

Computes empirical values identically to bioconductor/qvalue empPvals


@profile<EOL>def pi0est(p_values, lambda_ = np.arange(<NUM_LIT>,<NUM_LIT:1.0>,<NUM_LIT>), pi0_method = "<STR_LIT>", smooth_df = <NUM_LIT:3>, smooth_log_pi0 = False):

<EOL>p = np.array(p_values)<EOL>rm_na = np.isfinite(p)<EOL>p = p[rm_na]<EOL>m = len(p)<EOL>ll = <NUM_LIT:1><EOL>if isinstance(lambda_, np.ndarray ):<EOL><INDENT>ll = len(lambda_)<EOL>lambda_ = np.sort(lambda_)<EOL><DEDENT>if (min(p) < <NUM_LIT:0> or max(p) > <NUM_LIT:1>):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>elif (ll > <NUM_LIT:1> and ll < <NUM_LIT:4>):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>elif (np.min(lambda_) < <NUM_LIT:0> or np.max(lambda_) >= <NUM_LIT:1>):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>if (ll == <NUM_LIT:1>):<EOL><INDENT>pi0 = np.mean(p >= lambda_)/(<NUM_LIT:1> - lambda_)<EOL>pi0_lambda = pi0<EOL>pi0 = np.minimum(pi0, <NUM_LIT:1>)<EOL>pi0Smooth = False<EOL><DEDENT>else:<EOL><INDENT>pi0 = []<EOL>for l in lambda_:<EOL><INDENT>pi0.append(np.mean(p >= l)/(<NUM_LIT:1> - l))<EOL><DEDENT>pi0_lambda = pi0<EOL>if (pi0_method == "<STR_LIT>"):<EOL><INDENT>if smooth_log_pi0:<EOL><INDENT>pi0 = np.log(pi0)<EOL>spi0 = sp.interpolate.UnivariateSpline(lambda_, pi0, k=smooth_df)<EOL>pi0Smooth = np.exp(spi0(lambda_))<EOL><DEDENT>else:<EOL><INDENT>spi0 = sp.interpolate.UnivariateSpline(lambda_, pi0, k=smooth_df)<EOL>pi0Smooth = spi0(lambda_)<EOL><DEDENT>pi0 = np.minimum(pi0Smooth[ll-<NUM_LIT:1>],<NUM_LIT:1>)<EOL><DEDENT>elif (pi0_method == "<STR_LIT>"):<EOL><INDENT>minpi0 = np.percentile(pi0,<NUM_LIT:0.1>)<EOL>W = []<EOL>for l in lambda_:<EOL><INDENT>W.append(np.sum(p >= l))<EOL><DEDENT>mse = (np.array(W) / (np.power(m,<NUM_LIT:2>) * np.power((<NUM_LIT:1> - lambda_),<NUM_LIT:2>))) * (<NUM_LIT:1> - np.array(W) / m) + np.power((pi0 - minpi0),<NUM_LIT:2>)<EOL>pi0 = np.minimum(pi0[np.argmin(mse)],<NUM_LIT:1>)<EOL>pi0Smooth = False<EOL><DEDENT>else:<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT><DEDENT>if (pi0<=<NUM_LIT:0>):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>return {'<STR_LIT>': pi0, '<STR_LIT>': pi0_lambda, '<STR_LIT>': lambda_, '<STR_LIT>': pi0Smooth}<EOL>

Estimate pi0 according to bioconductor/qvalue


@profile<EOL>def lfdr(p_values, pi0, trunc = True, monotone = True, transf = "<STR_LIT>", adj = <NUM_LIT>, eps = np.power(<NUM_LIT>,-<NUM_LIT:8>)):

p = np.array(p_values)<EOL>lfdr_out = p<EOL>rm_na = np.isfinite(p)<EOL>p = p[rm_na]<EOL>if (min(p) < <NUM_LIT:0> or max(p) > <NUM_LIT:1>):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>elif (pi0 < <NUM_LIT:0> or pi0 > <NUM_LIT:1>):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>if (transf == "<STR_LIT>"):<EOL><INDENT>p = np.maximum(p, eps)<EOL>p = np.minimum(p, <NUM_LIT:1>-eps)<EOL>x = scipy.stats.norm.ppf(p, loc=<NUM_LIT:0>, scale=<NUM_LIT:1>)<EOL>bw = bw_nrd0(x)<EOL>myd = KDEUnivariate(x)<EOL>myd.fit(bw=adj*bw, gridsize = <NUM_LIT>)<EOL>splinefit = sp.interpolate.splrep(myd.support, myd.density)<EOL>y = sp.interpolate.splev(x, splinefit)<EOL>lfdr = pi0 * scipy.stats.norm.pdf(x) / y<EOL><DEDENT>elif (transf == "<STR_LIT>"):<EOL><INDENT>x = np.log((p + eps) / (<NUM_LIT:1> - p + eps))<EOL>bw = bw_nrd0(x)<EOL>myd = KDEUnivariate(x)<EOL>myd.fit(bw=adj*bw, gridsize = <NUM_LIT>)<EOL>splinefit = sp.interpolate.splrep(myd.support, myd.density)<EOL>y = sp.interpolate.splev(x, splinefit)<EOL>dx = np.exp(x) / np.power((<NUM_LIT:1> + np.exp(x)),<NUM_LIT:2>)<EOL>lfdr = (pi0 * dx) / y<EOL><DEDENT>else:<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>if (trunc):<EOL><INDENT>lfdr[lfdr > <NUM_LIT:1>] = <NUM_LIT:1><EOL><DEDENT>if (monotone):<EOL><INDENT>lfdr = lfdr[p.ravel().argsort()]<EOL>for i in range(<NUM_LIT:1>,len(x)):<EOL><INDENT>if (lfdr[i] < lfdr[i - <NUM_LIT:1>]):<EOL><INDENT>lfdr[i] = lfdr[i - <NUM_LIT:1>]<EOL><DEDENT><DEDENT>lfdr = lfdr[scipy.stats.rankdata(p,"<STR_LIT>")-<NUM_LIT:1>]<EOL><DEDENT>lfdr_out[rm_na] = lfdr<EOL>return lfdr_out<EOL>

Estimate local FDR / posterior error probability from p-values according to bioconductor/qvalue


cutoffs = df.cutoff.values<EOL>min_ = min(cutoffs)<EOL>max_ = max(cutoffs)<EOL>margin = (max_ - min_) * <NUM_LIT><EOL>sampled_cutoffs = np.linspace(min_ - margin, max_ + margin, num_cut_offs, dtype=np.float32)<EOL>ix = find_nearest_matches(np.float32(df.cutoff.values), sampled_cutoffs)<EOL>sampled_df = df.iloc[ix].copy()<EOL>sampled_df.cutoff = sampled_cutoffs<EOL>sampled_df.reset_index(inplace=True, drop=True)<EOL>return sampled_df<EOL>

Create artificial cutoff sample points from given range of cutoff
    values in df, number of sample points is 'num_cut_offs


@profile<EOL>def summary_err_table(df, qvalues=[<NUM_LIT:0>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT:0.1>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT>, <NUM_LIT:0.5>]):

qvalues = to_one_dim_array(qvalues)<EOL>ix = find_nearest_matches(np.float32(df.qvalue.values), qvalues)<EOL>df_sub = df.iloc[ix].copy()<EOL>for i_sub, (i0, i1) in enumerate(zip(ix, ix[<NUM_LIT:1>:])):<EOL><INDENT>if i1 == i0:<EOL><INDENT>df_sub.iloc[i_sub + <NUM_LIT:1>, :] = None<EOL><DEDENT><DEDENT>df_sub.qvalue = qvalues<EOL>df_sub.reset_index(inplace=True, drop=True)<EOL>return df_sub[['<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>','<STR_LIT>']]<EOL>

Summary error table for some typical q-values


@profile<EOL>def error_statistics(target_scores, decoy_scores, parametric, pfdr, pi0_lambda, pi0_method = "<STR_LIT>", pi0_smooth_df = <NUM_LIT:3>, pi0_smooth_log_pi0 = False, compute_lfdr = False, lfdr_trunc = True, lfdr_monotone = True, lfdr_transf = "<STR_LIT>", lfdr_adj = <NUM_LIT>, lfdr_eps = np.power(<NUM_LIT>,-<NUM_LIT:8>)):

target_scores = to_one_dim_array(target_scores)<EOL>target_scores = np.sort(target_scores[~np.isnan(target_scores)])<EOL>decoy_scores = to_one_dim_array(decoy_scores)<EOL>decoy_scores = np.sort(decoy_scores[~np.isnan(decoy_scores)])<EOL>if parametric:<EOL><INDENT>target_pvalues = pnorm(target_scores, decoy_scores)<EOL><DEDENT>else:<EOL><INDENT>target_pvalues = pemp(target_scores, decoy_scores)<EOL><DEDENT>pi0 = pi0est(target_pvalues, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0)<EOL>target_qvalues = qvalue(target_pvalues, pi0['<STR_LIT>'], pfdr)<EOL>metrics = stat_metrics(target_pvalues, pi0['<STR_LIT>'], pfdr)<EOL>error_stat = pd.DataFrame({'<STR_LIT>': target_scores, '<STR_LIT>': target_pvalues, '<STR_LIT>': target_qvalues, '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>'], '<STR_LIT>': metrics['<STR_LIT>']})<EOL>if compute_lfdr:<EOL><INDENT>error_stat['<STR_LIT>'] = lfdr(target_pvalues, pi0['<STR_LIT>'], lfdr_trunc, lfdr_monotone, lfdr_transf, lfdr_adj, lfdr_eps)<EOL><DEDENT>return error_stat, pi0<EOL>

Takes list of decoy and target scores and creates error statistics for target values


def find_cutoff(tt_scores, td_scores, cutoff_fdr, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0):

error_stat, pi0 = error_statistics(tt_scores, td_scores, parametric, pfdr, pi0_lambda, pi0_method, pi0_smooth_df, pi0_smooth_log_pi0, False)<EOL>if not len(error_stat):<EOL><INDENT>raise click.ClickException("<STR_LIT>")<EOL><DEDENT>i0 = (error_stat.qvalue - cutoff_fdr).abs().idxmin()<EOL>cutoff = error_stat.iloc[i0]["<STR_LIT>"]<EOL>return cutoff<EOL>

Finds cut off target score for specified false discovery rate fdr


if not settings.UNFRIENDLY_ENABLE_FILTER:<EOL><INDENT>return value<EOL><DEDENT>kwargs = {<EOL>'<STR_LIT:key>': encrypt(value,<EOL>settings.UNFRIENDLY_SECRET,<EOL>settings.UNFRIENDLY_IV,<EOL>checksum=settings.UNFRIENDLY_ENFORCE_CHECKSUM),<EOL>}<EOL>if juice:<EOL><INDENT>kwargs['<STR_LIT>'] = slugify(juice)<EOL><DEDENT>return reverse('<STR_LIT>', kwargs=kwargs)<EOL>

Template filter that obfuscates whatever text it is applied to. The text is
supposed to be a URL, but it will obfuscate anything.

Usage:
    Extremely unfriendly URL:
    {{ "/my-secret-path/"|obfuscate }}

    Include some SEO juice:
    {{ "/my-secret-path/"|obfuscate:"some SEO friendly text" }}


def deobfuscate(request, key, juice=None):

try:<EOL><INDENT>url = decrypt(str(key),<EOL>settings.UNFRIENDLY_SECRET,<EOL>settings.UNFRIENDLY_IV,<EOL>checksum=settings.UNFRIENDLY_ENFORCE_CHECKSUM)<EOL><DEDENT>except (CheckSumError, InvalidKeyError):<EOL><INDENT>return HttpResponseNotFound()<EOL><DEDENT>try:<EOL><INDENT>url = url.decode('<STR_LIT:utf-8>')<EOL><DEDENT>except UnicodeDecodeError:<EOL><INDENT>return HttpResponseNotFound()<EOL><DEDENT>url_parts = urlparse(unquote(url))<EOL>path = url_parts.path<EOL>query = url_parts.query<EOL>try:<EOL><INDENT>view, args, kwargs = resolve(path)<EOL><DEDENT>except Resolver404:<EOL><INDENT>return HttpResponseNotFound()<EOL><DEDENT>environ = request.environ.copy()<EOL>environ['<STR_LIT>'] = path[len(environ['<STR_LIT>']):]<EOL>environ['<STR_LIT>'] = query<EOL>patched_request = request.__class__(environ)<EOL>missing_items = set(dir(request)) - set(dir(patched_request))<EOL>while missing_items:<EOL><INDENT>missing_item = missing_items.pop()<EOL>patched_request.__setattr__(missing_item,<EOL>request.__getattribute__(missing_item))<EOL><DEDENT>patched_request.META['<STR_LIT>'] = True<EOL>response = view(patched_request, *args, **kwargs)<EOL>if juice and not response.has_header('<STR_LIT>'):<EOL><INDENT>response['<STR_LIT>'] = '<STR_LIT>' % juice<EOL><DEDENT>return response<EOL>

Deobfuscates the URL and returns HttpResponse from source view.
SEO juice is mostly ignored as it is intended for display purposes only.


def _lazysecret(secret, blocksize=<NUM_LIT:32>, padding='<STR_LIT:}>'):

if not len(secret) in (<NUM_LIT:16>, <NUM_LIT>, <NUM_LIT:32>):<EOL><INDENT>return secret + (blocksize - len(secret)) * padding<EOL><DEDENT>return secret<EOL>

Pads secret if not legal AES block size (16, 24, 32)


if not isinstance(plaintext, six.binary_type):<EOL><INDENT>plaintext = six.b(plaintext)<EOL><DEDENT>return (zlib.crc32(plaintext) % <NUM_LIT>) & <NUM_LIT><EOL>

Generates crc32. Modulo keep the value within int range.


return struct.pack('<STR_LIT:i>', _crc(plaintext))<EOL>


def encrypt(plaintext, secret, inital_vector, checksum=True, lazy=True):

if not isinstance(plaintext, six.binary_type):<EOL><INDENT>plaintext = six.b(plaintext)<EOL><DEDENT>secret = _lazysecret(secret) if lazy else secret<EOL>encobj = AES.new(secret, AES.MODE_CFB, inital_vector)<EOL>if checksum:<EOL><INDENT>packed = _pack_crc(plaintext)<EOL>plaintext += base64.urlsafe_b64encode(packed)<EOL><DEDENT>encoded = base64.urlsafe_b64encode(encobj.encrypt(plaintext))<EOL>if isinstance(plaintext, six.binary_type):<EOL><INDENT>encoded = encoded.decode()<EOL><DEDENT>return encoded.replace('<STR_LIT:=>', '<STR_LIT>')<EOL>

Encrypts plaintext with secret
    plaintext      - content to encrypt
    secret         - secret to encrypt plaintext
    inital_vector  - initial vector
    lazy           - pad secret if less than legal blocksize (default: True)
    checksum       - attach crc32 byte encoded (default: True)
    returns ciphertext


def decrypt(ciphertext, secret, inital_vector, checksum=True, lazy=True):

secret = _lazysecret(secret) if lazy else secret<EOL>encobj = AES.new(secret, AES.MODE_CFB, inital_vector)<EOL>try:<EOL><INDENT>padded = ciphertext + ('<STR_LIT:=>' * (len(ciphertext) % <NUM_LIT:4>))<EOL>decoded = base64.urlsafe_b64decode(str(padded))<EOL>plaintext = encobj.decrypt(decoded)<EOL><DEDENT>except (TypeError, binascii.Error):<EOL><INDENT>raise InvalidKeyError("<STR_LIT>")<EOL><DEDENT>if checksum:<EOL><INDENT>try:<EOL><INDENT>crc, plaintext = (base64.urlsafe_b64decode(<EOL>plaintext[-<NUM_LIT:8>:]), plaintext[:-<NUM_LIT:8>])<EOL><DEDENT>except (TypeError, binascii.Error):<EOL><INDENT>raise CheckSumError("<STR_LIT>")<EOL><DEDENT>if not crc == _pack_crc(plaintext):<EOL><INDENT>raise CheckSumError("<STR_LIT>")<EOL><DEDENT><DEDENT>return plaintext<EOL>

Decrypts ciphertext with secret
    ciphertext     - encrypted content to decrypt
    secret         - secret to decrypt ciphertext
    inital_vector  - initial vector
    lazy           - pad secret if less than legal blocksize (default: True)
    checksum       - verify crc32 byte encoded checksum (default: True)
    returns plaintext


return event_number_1 != event_number_2<EOL>

Detect a new event by checking if the event number of the actual hit is the actual event number


if array.shape[<NUM_LIT:0>] > hit["<STR_LIT>"] and array.shape[<NUM_LIT:1>] > hit["<STR_LIT>"]:<EOL><INDENT>return array[hit["<STR_LIT>"], hit["<STR_LIT>"]]<EOL><DEDENT>else:<EOL><INDENT>return False<EOL><DEDENT>

Checks whether a hit (column/row) is masked or not. Array is 2D array with boolean elements corresponding to pixles indicating whether a pixel is disabled or not.


@njit()<EOL>def _finish_cluster(hits, clusters, cluster_size, cluster_hit_indices, cluster_index, cluster_id, charge_correction, noisy_pixels, disabled_pixels):

cluster_charge = <NUM_LIT:0><EOL>max_cluster_charge = -<NUM_LIT:1><EOL>total_weighted_column = <NUM_LIT:0><EOL>total_weighted_row = <NUM_LIT:0><EOL>for i in range(cluster_size):<EOL><INDENT>hit_index = cluster_hit_indices[i]<EOL>if hits[hit_index]['<STR_LIT>'] > max_cluster_charge:<EOL><INDENT>seed_hit_index = hit_index<EOL>max_cluster_charge = hits[hit_index]['<STR_LIT>']<EOL><DEDENT>hits[hit_index]['<STR_LIT>'] = <NUM_LIT:0><EOL>hits[hit_index]['<STR_LIT>'] = cluster_size<EOL>total_weighted_column += hits[hit_index]['<STR_LIT>'] * (hits[hit_index]['<STR_LIT>'] + charge_correction)<EOL>total_weighted_row += hits[hit_index]['<STR_LIT>'] * (hits[hit_index]['<STR_LIT>'] + charge_correction)<EOL>cluster_charge += hits[hit_index]['<STR_LIT>']<EOL>hits[hit_index]['<STR_LIT>'] = cluster_id<EOL><DEDENT>hits[seed_hit_index]['<STR_LIT>'] = <NUM_LIT:1><EOL>clusters[cluster_index]["<STR_LIT>"] = cluster_id<EOL>clusters[cluster_index]["<STR_LIT>"] = cluster_size<EOL>clusters[cluster_index]["<STR_LIT>"] = cluster_charge<EOL>clusters[cluster_index]['<STR_LIT>'] = hits[seed_hit_index]['<STR_LIT>']<EOL>clusters[cluster_index]['<STR_LIT>'] = hits[seed_hit_index]['<STR_LIT>']<EOL>clusters[cluster_index]['<STR_LIT>'] = float(total_weighted_column) / (cluster_charge + cluster_size * charge_correction)<EOL>clusters[cluster_index]['<STR_LIT>'] = float(total_weighted_row) / (cluster_charge + cluster_size * charge_correction)<EOL>_end_of_cluster_function(<EOL>hits=hits,<EOL>clusters=clusters,<EOL>cluster_size=cluster_size,<EOL>cluster_hit_indices=cluster_hit_indices,<EOL>cluster_index=cluster_index,<EOL>cluster_id=cluster_id,<EOL>charge_correction=charge_correction,<EOL>noisy_pixels=noisy_pixels,<EOL>disabled_pixels=disabled_pixels,<EOL>seed_hit_index=seed_hit_index)<EOL>

Set hit and cluster information of the cluster (e.g. number of hits in the cluster (cluster_size), total cluster charge (charge), ...).


@njit()<EOL>def _finish_event(hits, clusters, start_event_hit_index, stop_event_hit_index, start_event_cluster_index, stop_event_cluster_index):

for hit_index in range(start_event_hit_index, stop_event_hit_index):<EOL><INDENT>hits[hit_index]['<STR_LIT>'] = stop_event_cluster_index - start_event_cluster_index<EOL><DEDENT>for cluster_index in range(start_event_cluster_index, stop_event_cluster_index):<EOL><INDENT>clusters[cluster_index]['<STR_LIT>'] = hits[start_event_hit_index]['<STR_LIT>']<EOL><DEDENT>_end_of_event_function(<EOL>hits=hits,<EOL>clusters=clusters,<EOL>start_event_hit_index=start_event_hit_index,<EOL>stop_event_hit_index=stop_event_hit_index,<EOL>start_event_cluster_index=start_event_cluster_index,<EOL>stop_event_cluster_index=stop_event_cluster_index)<EOL>

Set hit and cluster information of the event (e.g. number of cluster in the event (n_cluster), ...).


@njit()<EOL>def _hit_ok(hit, min_hit_charge, max_hit_charge):

<EOL>if hit['<STR_LIT>'] < min_hit_charge:<EOL><INDENT>return False<EOL><DEDENT>if max_hit_charge != <NUM_LIT:0> and hit['<STR_LIT>'] > max_hit_charge:<EOL><INDENT>return False<EOL><DEDENT>return True<EOL>

Check if given hit is withing the limits.


hit['<STR_LIT>'] = cluster_id<EOL>hit['<STR_LIT>'] = <NUM_LIT:0><EOL>hit['<STR_LIT>'] = <NUM_LIT:0><EOL>


@njit()<EOL>def _set_1d_array(array, value, size=-<NUM_LIT:1>):

if size >= <NUM_LIT:0>:<EOL><INDENT>for i in range(size):<EOL><INDENT>array[i] = value<EOL><DEDENT><DEDENT>else:<EOL><INDENT>for i in range(array.shape[<NUM_LIT:0>]):<EOL><INDENT>array[i] = value<EOL><DEDENT><DEDENT>

Set array elemets to value for given number of elements (if size is negative number set all elements to value).


@njit()<EOL>def _is_in_max_difference(value_1, value_2, max_difference):

if value_1 <= value_2:<EOL><INDENT>return value_2 - value_1 <= max_difference<EOL><DEDENT>return value_1 - value_2 <= max_difference<EOL>

Helper function to determine the difference of two values that can be np.uints. Works in python and numba mode.
    Circumvents numba bug #1653