BaseStatsCollector.getStats(self, stats)<EOL>sortedNumberList = sorted(self.valueList)<EOL>listLength = len(sortedNumberList)<EOL>min = sortedNumberList[<NUM_LIT:0>]<EOL>max = sortedNumberList[-<NUM_LIT:1>]<EOL>mean = numpy.mean(self.valueList)<EOL>median = sortedNumberList[int(<NUM_LIT:0.5>*listLength)]<EOL>percentile1st = sortedNumberList[int(<NUM_LIT>*listLength)]<EOL>percentile99th = sortedNumberList[int(<NUM_LIT>*listLength)]<EOL>differenceList =[(cur - prev) for prev, cur in zip(list(self.valueSet)[:-<NUM_LIT:1>],<EOL>list(self.valueSet)[<NUM_LIT:1>:])]<EOL>if min > max:<EOL><INDENT>print(self.fieldname, min, max, '<STR_LIT>')<EOL><DEDENT>meanResolution = numpy.mean(differenceList)<EOL>stats[self.fieldname]['<STR_LIT>'] = min<EOL>stats[self.fieldname]['<STR_LIT>'] = max<EOL>stats[self.fieldname]['<STR_LIT>'] = mean<EOL>stats[self.fieldname]['<STR_LIT>'] = median<EOL>stats[self.fieldname]['<STR_LIT>'] = percentile1st<EOL>stats[self.fieldname]['<STR_LIT>'] = percentile99th<EOL>stats[self.fieldname]['<STR_LIT>'] = meanResolution<EOL>passData = True<EOL>if passData:<EOL><INDENT>stats[self.fieldname]['<STR_LIT:data>'] = self.valueList<EOL><DEDENT>if VERBOSITY > <NUM_LIT:2>:<EOL><INDENT>print('<STR_LIT>')<EOL>print("<STR_LIT>")<EOL>print("<STR_LIT>", min)<EOL>print("<STR_LIT>", max)<EOL>print("<STR_LIT>", mean)<EOL>print("<STR_LIT>", median)<EOL>print("<STR_LIT>", percentile1st)<EOL>print("<STR_LIT>", percentile99th)<EOL>print('<STR_LIT>')<EOL>print("<STR_LIT>")<EOL>print("<STR_LIT>", meanResolution)<EOL><DEDENT>if VERBOSITY > <NUM_LIT:3>:<EOL><INDENT>print('<STR_LIT>')<EOL>print("<STR_LIT>")<EOL>counts, bins = numpy.histogram(self.valueList, new=True)<EOL>print("<STR_LIT>", counts.tolist())<EOL>print("<STR_LIT>", bins.tolist())<EOL><DEDENT>

Override of getStats()  in BaseStatsCollector

            stats: A dictionary where all the stats are
            outputted


for field, meta in self.filterDict.iteritems():<EOL><INDENT>index = meta['<STR_LIT:index>']<EOL>categories = meta['<STR_LIT>']<EOL>for category in categories:<EOL><INDENT>if not record:<EOL><INDENT>continue<EOL><DEDENT>if record[index].find(category) != -<NUM_LIT:1>:<EOL><INDENT>'''<STR_LIT>'''<EOL>return True<EOL><DEDENT><DEDENT><DEDENT>return False<EOL>

Returns True if the record matches any of the provided filters


return fieldDataType in cls._ALL<EOL>

Check a candidate value whether it's one of the valid field data types

        :param fieldDataType: (string) candidate field data type
        :returns: True if the candidate value is a legitimate field data type value;
                  False if not


if s is None:<EOL><INDENT>return '<STR_LIT>'<EOL><DEDENT>assert isinstance(s, basestring),"<STR_LIT>" % (basestring, type(s), s)<EOL>s = s.replace('<STR_LIT:\\>', '<STR_LIT>')<EOL>s = s.replace('<STR_LIT:\n>', '<STR_LIT>')<EOL>s = s.replace('<STR_LIT:\t>', '<STR_LIT>')<EOL>s = s.replace('<STR_LIT:U+002C>', '<STR_LIT:\t>')<EOL>return s<EOL>

Escape commas, tabs, newlines and dashes in a string

Commas are encoded as tabs.

:param s: (string) to escape
:returns: (string) escaped string


return t.strftime(DATETIME_FORMATS[<NUM_LIT:2>])<EOL>

Turns a datetime object into a string ignoring milliseconds.

:param t: (datetime.datetime)
:return: (string) in default format (see
         :const:`~nupic.data.utils.DATETIME_FORMATS` [2])


if i.strip() == '<STR_LIT:None>' or i.strip() == '<STR_LIT>':<EOL><INDENT>return None<EOL><DEDENT>return int(i)<EOL>

Tries to convert input to a int input or returns ``None``.

:param f: (object) thing to convert to a int
:return: (int or ``None``)


l = s.lower()<EOL>if l in ("<STR_LIT:true>", "<STR_LIT:t>", "<STR_LIT:1>"):<EOL><INDENT>return True<EOL><DEDENT>if l in ("<STR_LIT:false>", "<STR_LIT:f>", "<STR_LIT:0>"):<EOL><INDENT>return False<EOL><DEDENT>raise Exception("<STR_LIT>" % s)<EOL>

String to boolean

:param s: (string)
:return: (bool)


if f == '<STR_LIT:None>':<EOL><INDENT>return None<EOL><DEDENT>return float(f)<EOL>

Tries to convert input to a float input or returns ``None``.

:param f: (object) thing to convert to a float
:return: (float or ``None``)


if bit >= self._n:<EOL><INDENT>raise IndexError("<STR_LIT>")<EOL><DEDENT>numbers = set()<EOL>for index, pattern in self._patterns.items():<EOL><INDENT>if bit in pattern:<EOL><INDENT>numbers.add(index)<EOL><DEDENT><DEDENT>return numbers<EOL>

Return the set of pattern numbers that match a bit.

@param bit (int) Index of bit

@return (set) Indices of numbers


def prettyPrintPattern(self, bits, verbosity=<NUM_LIT:1>):

numberMap = self.numberMapForBits(bits)<EOL>text = "<STR_LIT>"<EOL>numberList = []<EOL>numberItems = sorted(iter(numberMap.items()),<EOL>key=lambda number_bits: len(number_bits[<NUM_LIT:1>]),<EOL>reverse=True)<EOL>for number, bits in numberItems:<EOL><INDENT>if verbosity > <NUM_LIT:2>:<EOL><INDENT>strBits = [str(n) for n in bits]<EOL>numberText = "<STR_LIT>".format(number, "<STR_LIT:U+002C>".join(strBits))<EOL><DEDENT>elif verbosity > <NUM_LIT:1>:<EOL><INDENT>numberText = "<STR_LIT>".format(number, len(bits))<EOL><DEDENT>else:<EOL><INDENT>numberText = str(number)<EOL><DEDENT>numberList.append(numberText)<EOL><DEDENT>text += "<STR_LIT>".format("<STR_LIT:U+002CU+0020>".join(numberList))<EOL>return text<EOL>

Pretty print a pattern.

@param bits      (set) Indices of on bits
@param verbosity (int) Verbosity level

@return (string) Pretty-printed text


numberMap = dict()<EOL>for bit in bits:<EOL><INDENT>numbers = self.numbersForBit(bit)<EOL>for number in numbers:<EOL><INDENT>if not number in numberMap:<EOL><INDENT>numberMap[number] = set()<EOL><DEDENT>numberMap[number].add(bit)<EOL><DEDENT><DEDENT>return numberMap<EOL>

Return a map from number to matching on bits,
for all numbers that match a set of bits.

@param bits (set) Indices of bits

@return (dict) Mapping from number => on bits.


candidates = np.array(list(range(self._n)), np.uint32)<EOL>for i in range(self._num):<EOL><INDENT>self._random.shuffle(candidates)<EOL>pattern = candidates[<NUM_LIT:0>:self._getW()]<EOL>self._patterns[i] = set(pattern)<EOL><DEDENT>


w = self._w<EOL>if type(w) is list:<EOL><INDENT>return w[self._random.getUInt32(len(w))]<EOL><DEDENT>else:<EOL><INDENT>return w<EOL><DEDENT>

Gets a value of `w` for use in generating a pattern.


sequence = []<EOL>for number in numbers:<EOL><INDENT>if number == None:<EOL><INDENT>sequence.append(number)<EOL><DEDENT>else:<EOL><INDENT>pattern = self.patternMachine.get(number)<EOL>sequence.append(pattern)<EOL><DEDENT><DEDENT>return sequence<EOL>

Generate a sequence from a list of numbers.

Note: Any `None` in the list of numbers is considered a reset.

@param numbers (list) List of numbers

@return (list) Generated sequence


def generateNumbers(self, numSequences, sequenceLength, sharedRange=None):

numbers = []<EOL>if sharedRange:<EOL><INDENT>sharedStart, sharedEnd = sharedRange<EOL>sharedLength = sharedEnd - sharedStart<EOL>sharedNumbers = range(numSequences * sequenceLength,<EOL>numSequences * sequenceLength + sharedLength)<EOL><DEDENT>for i in xrange(numSequences):<EOL><INDENT>start = i * sequenceLength<EOL>newNumbers = np.array(range(start, start + sequenceLength), np.uint32)<EOL>self._random.shuffle(newNumbers)<EOL>newNumbers = list(newNumbers)<EOL>if sharedRange is not None:<EOL><INDENT>newNumbers[sharedStart:sharedEnd] = sharedNumbers<EOL><DEDENT>numbers += newNumbers<EOL>numbers.append(None)<EOL><DEDENT>return numbers<EOL>

@param numSequences   (int)   Number of sequences to return,
                              separated by None
@param sequenceLength (int)   Length of each sequence
@param sharedRange    (tuple) (start index, end index) indicating range of
                              shared subsequence in each sequence
                              (None if no shared subsequences)
@return (list) Numbers representing sequences


for v in values:<EOL><INDENT>self.addValue(v)<EOL><DEDENT>


numRecords = self.fields[<NUM_LIT:0>].numRecords<EOL>assert (all(field.numRecords==numRecords for field in self.fields))<EOL>import csv<EOL>with open(path+'<STR_LIT>', '<STR_LIT:wb>') as f:<EOL><INDENT>writer = csv.writer(f)<EOL>writer.writerow(self.getAllFieldNames())<EOL>writer.writerow(self.getAllDataTypes())<EOL>writer.writerow(self.getAllFlags())<EOL>writer.writerows(self.getAllRecords())<EOL><DEDENT>if self.verbosity><NUM_LIT:0>:<EOL><INDENT>print('<STR_LIT>', numRecords,'<STR_LIT>',path,'<STR_LIT>')<EOL><DEDENT>

Export all the records into a csv file in numenta format.

        Example header format:
        fieldName1    fieldName2    fieldName3
        date          string        float
        T             S

        Parameters:
        --------------------------------------------------------------------
        path:      Relative path of the file to which the records are to be exported


def addValuesToField(self, i, numValues):

assert(len(self.fields)>i)<EOL>values = [self.addValueToField(i) for n in range(numValues)]<EOL>return values<EOL>


dataTypes = [field.dataType for field in self.fields]<EOL>return dataTypes<EOL>


def encodeRecord(self, record, toBeAdded=True):

encoding=[self.fields[i].encodeValue(record[i], toBeAdded) for i inrange(len(self.fields))]<EOL>return encoding<EOL>

Encode a record as a sparse distributed representation
        Parameters:
        --------------------------------------------------------------------
        record:        Record to be encoded
        toBeAdded:     Whether the encodings corresponding to the record are added to
                       the corresponding fields


assert all(x in field for x in ['<STR_LIT:name>', '<STR_LIT>', '<STR_LIT>'] for fieldin fieldsInfo)<EOL>for spec in fieldsInfo:<EOL><INDENT>self.addField(spec.pop('<STR_LIT:name>'), spec.pop('<STR_LIT>'), spec.pop('<STR_LIT>'))<EOL><DEDENT>

Add multiple fields to the dataset.
        Parameters:
        -------------------------------------------------------------------
        fieldsInfo:       A list of dictionaries, containing a field name, specs for
                          the data classes and encoder params for the corresponding
                          field.


def addValueToField(self, i, value=None):

assert(len(self.fields)>i)<EOL>if value is None:<EOL><INDENT>value = self.fields[i].dataClass.getNext()<EOL>self.fields[i].addValue(value)<EOL>return value<EOL><DEDENT>else: self.fields[i].addValue(value)<EOL>


def addField(self, name, fieldParams, encoderParams):

assert fieldParams is not None and'<STR_LIT:type>' in fieldParams<EOL>dataClassName = fieldParams.pop('<STR_LIT:type>')<EOL>try:<EOL><INDENT>dataClass=eval(dataClassName)(fieldParams)<EOL><DEDENT>except TypeError as e:<EOL><INDENT>print(("<STR_LIT>"<EOL>"<STR_LIT>"<EOL>"<STR_LIT>" % (dataClass, fieldParams)))<EOL>raise<EOL><DEDENT>encoderParams['<STR_LIT>']=dataClass<EOL>encoderParams['<STR_LIT>']=dataClassName<EOL>fieldIndex = self.defineField(name, encoderParams)<EOL>

Add a single field to the dataset.
        Parameters:
        -------------------------------------------------------------------
        name:             The user-specified name of the field
        fieldSpec:        A list of one or more dictionaries specifying parameters
                          to be used for dataClass initialization. Each dict must
                          contain the key 'type' that specifies a distribution for
                          the values in this field
        encoderParams:    Parameters for the field encoder


self.values.append(value)<EOL>self.numRecords+=<NUM_LIT:1><EOL>


n = sum([field.n for field in self.fields])<EOL>return n<EOL>

Returns the cumulative n for all the fields in the dataset


self.name=name<EOL>self.n, self.w = (<NUM_LIT:100>, <NUM_LIT:15>)<EOL>self.encoderType,self.dataType,self.dataClassName  = (None, None, None)<EOL>self.flag='<STR_LIT>'<EOL>self.isPredictedField=False<EOL>if encoderSpec is not None:<EOL><INDENT>if '<STR_LIT:n>' in encoderSpec: self.n = encoderSpec.pop('<STR_LIT:n>')<EOL>if '<STR_LIT:w>' in encoderSpec: self.w = encoderSpec.pop('<STR_LIT:w>')<EOL>if '<STR_LIT>' in encoderSpec: self.flag = encoderSpec.pop('<STR_LIT>')<EOL>if '<STR_LIT>' in encoderSpec: self.isPredictedField= encoderSpec.pop('<STR_LIT>')<EOL>if '<STR_LIT>' in encoderSpec: self.dataClass= encoderSpec.pop('<STR_LIT>')<EOL>if '<STR_LIT>' in encoderSpec: self.dataClassName= encoderSpec.pop('<STR_LIT>')<EOL>if '<STR_LIT>' in encoderSpec: self.dataType = encoderSpec.pop('<STR_LIT>')<EOL>if '<STR_LIT>' in encoderSpec: self.encoderType= encoderSpec.pop('<STR_LIT>')<EOL><DEDENT>if self.dataType is None and self.encoderType is None:<EOL><INDENT>raise RuntimeError('<STR_LIT>')<EOL><DEDENT>assert(self.dataType is not None or self.encoderType is not None)<EOL>if self.dataType is None or self.encoderType is None:<EOL><INDENT>self._setTypes(encoderSpec)<EOL><DEDENT>self._initializeEncoders(encoderSpec)<EOL>self.encodings=[]<EOL>self.values=[]<EOL>self.numRecords=<NUM_LIT:0><EOL>self.numEncodings=<NUM_LIT:0><EOL>

Initialize a field with various parameters such as n, w, flag, dataType,
        encoderType, and tag predicted field.


def defineField(self, name, encoderParams=None):

self.fields.append(_field(name, encoderParams))<EOL>return len(self.fields)-<NUM_LIT:1><EOL>

Initialize field using relevant encoder parameters.
        Parameters:
        -------------------------------------------------------------------
        name:                 Field name
        encoderParams:        Parameters for the encoder.

        Returns the index of the field


def encodeAllRecords(self, records=None, toBeAdded=True):

if records is None:<EOL><INDENT>records = self.getAllRecords()<EOL><DEDENT>if self.verbosity><NUM_LIT:0>: print('<STR_LIT>', len(records), '<STR_LIT>')<EOL>encodings = [self.encodeRecord(record, toBeAdded) for record in records]<EOL>return encodings<EOL>

Encodes a list of records.
        Parameters:
        --------------------------------------------------------------------
        records:      One or more records. (i,j)th element of this 2D array
                      specifies the value at field j of record i.
                      If unspecified, records previously generated and stored are
                      used.
        toBeAdded:    Whether the encodings corresponding to the record are added to
                      the corresponding fields


names = [field.name for field in self.fields]<EOL>return names<EOL>


def copy(reader, writer, start, stop, insertLocation=None, tsCol=None):

assert stop >= start<EOL>startRows = []<EOL>copyRows = []<EOL>ts = None<EOL>inc = None<EOL>if tsCol is None:<EOL><INDENT>tsCol = reader.getTimestampFieldIdx()<EOL><DEDENT>for i, row in enumerate(reader):<EOL><INDENT>if ts is None:<EOL><INDENT>ts = row[tsCol]<EOL><DEDENT>elif inc is None:<EOL><INDENT>inc = row[tsCol] - ts<EOL><DEDENT>if i >= start and i <= stop:<EOL><INDENT>copyRows.append(row)<EOL><DEDENT>startRows.append(row)<EOL><DEDENT>if insertLocation is None:<EOL><INDENT>insertLocation = stop + <NUM_LIT:1><EOL><DEDENT>startRows[insertLocation:insertLocation] = copyRows<EOL>for row in startRows:<EOL><INDENT>row[tsCol] = ts<EOL>writer.appendRecord(row)<EOL>ts += inc<EOL><DEDENT>

Copies a range of values to a new location in the data set.

    Args:
      reader: A FileRecordStream object with input data.
      writer: A FileRecordStream object to write output data to.
      start: The first row in the range to copy.
      stop: The last row in the range to copy.
      insertLocation: The location to insert the copied range. If not specified,
          the range is inserted immediately following itself.


def sample(reader, writer, n, start=None, stop=None, tsCol=None,<EOL>writeSampleOnly=True):

rows = list(reader)<EOL>if tsCol is not None:<EOL><INDENT>ts = rows[<NUM_LIT:0>][tsCol]<EOL>inc = rows[<NUM_LIT:1>][tsCol] - ts<EOL><DEDENT>if start is None:<EOL><INDENT>start = <NUM_LIT:0><EOL><DEDENT>if stop is None:<EOL><INDENT>stop = len(rows) - <NUM_LIT:1><EOL><DEDENT>initialN = stop - start + <NUM_LIT:1><EOL>numDeletes =  initialN - n<EOL>for i in range(numDeletes):<EOL><INDENT>delIndex = random.randint(start, stop - i)<EOL>del rows[delIndex]<EOL><DEDENT>if writeSampleOnly:<EOL><INDENT>rows = rows[start:start + n]<EOL><DEDENT>if tsCol is not None:<EOL><INDENT>ts = rows[<NUM_LIT:0>][tsCol]<EOL><DEDENT>for row in rows:<EOL><INDENT>if tsCol is not None:<EOL><INDENT>row[tsCol] = ts<EOL>ts += inc<EOL><DEDENT>writer.appendRecord(row)<EOL><DEDENT>

Samples n rows.

    Args:
      reader: A FileRecordStream object with input data.
      writer: A FileRecordStream object to write output data to.
      n: The number of elements to sample.
      start: The first row in the range to sample from.
      stop: The last row in the range to sample from.
      tsCol: If specified, the timestamp column to update.
      writeSampleOnly: If False, the rows before start are written before the
          sample and the rows after stop are written after the sample.


A distribution is a set of values with certain statistical properties

        Methods/properties that must be implemented by subclasses
        - getNext() -- Returns the next value for the distribution
        - getData(n) -- Returns n values for the distribution
        - getDescription() -- Returns a dict of parameters pertinent to the
          distribution, if any as well as state variables.


records = [self.getNext() for x in range(n)]<EOL>return records<EOL>

Returns the next n values for the distribution as a list.


hasReset = self.resetFieldName is not None<EOL>hasSequenceId = self.sequenceIdFieldName is not None<EOL>if hasReset and not hasSequenceId:<EOL><INDENT>self._sequenceInfoType = self.SEQUENCEINFO_RESET_ONLY<EOL>self._prevSequenceId = <NUM_LIT:0><EOL><DEDENT>elif not hasReset and hasSequenceId:<EOL><INDENT>self._sequenceInfoType = self.SEQUENCEINFO_SEQUENCEID_ONLY<EOL>self._prevSequenceId = None<EOL><DEDENT>elif hasReset and hasSequenceId:<EOL><INDENT>self._sequenceInfoType = self.SEQUENCEINFO_BOTH<EOL><DEDENT>else:<EOL><INDENT>self._sequenceInfoType = self.SEQUENCEINFO_NONE<EOL><DEDENT>

Figure out whether reset, sequenceId,
        both or neither are present in the data.
        Compute once instead of every time.

        Taken from filesource.py


inferencesToWrite = {}<EOL>if self._inferenceBuffer is None:<EOL><INDENT>maxDelay = InferenceElement.getMaxDelay(modelResult.inferences)<EOL>self._inferenceBuffer = collections.deque(maxlen=maxDelay + <NUM_LIT:1>)<EOL><DEDENT>self._inferenceBuffer.appendleft(copy.deepcopy(modelResult.inferences))<EOL>for inferenceElement, inference in modelResult.inferences.iteritems():<EOL><INDENT>if isinstance(inference, dict):<EOL><INDENT>inferencesToWrite[inferenceElement] = {}<EOL>for key, _ in inference.iteritems():<EOL><INDENT>delay = InferenceElement.getTemporalDelay(inferenceElement, key)<EOL>if len(self._inferenceBuffer) > delay:<EOL><INDENT>prevInference = self._inferenceBuffer[delay][inferenceElement][key]<EOL>inferencesToWrite[inferenceElement][key] = prevInference<EOL><DEDENT>else:<EOL><INDENT>inferencesToWrite[inferenceElement][key] = None<EOL><DEDENT><DEDENT><DEDENT>else:<EOL><INDENT>delay = InferenceElement.getTemporalDelay(inferenceElement)<EOL>if len(self._inferenceBuffer) > delay:<EOL><INDENT>inferencesToWrite[inferenceElement] = (<EOL>self._inferenceBuffer[delay][inferenceElement])<EOL><DEDENT>else:<EOL><INDENT>if type(inference) in (list, tuple):<EOL><INDENT>inferencesToWrite[inferenceElement] = [None] * len(inference)<EOL><DEDENT>else:<EOL><INDENT>inferencesToWrite[inferenceElement] = None<EOL><DEDENT><DEDENT><DEDENT><DEDENT>shiftedResult = ModelResult(rawInput=modelResult.rawInput,<EOL>sensorInput=modelResult.sensorInput,<EOL>inferences=inferencesToWrite,<EOL>metrics=modelResult.metrics,<EOL>predictedFieldIdx=modelResult.predictedFieldIdx,<EOL>predictedFieldName=modelResult.predictedFieldName)<EOL>return shiftedResult<EOL>

Shift the model result and return the new instance.

        Queues up the T(i+1) prediction value and emits a T(i)
        input/prediction pair, if possible. E.g., if the previous T(i-1)
        iteration was learn-only, then we would not have a T(i) prediction in our
        FIFO and would not be able to emit a meaningful input/prediction pair.

        :param modelResult: A :class:`~.nupic.frameworks.opf.opf_utils.ModelResult`
                            instance to shift.
        :return: A :class:`~.nupic.frameworks.opf.opf_utils.ModelResult` instance that
                 has been shifted


with open(inputFilePath) as fileObj:<EOL><INDENT>value = json.load(fileObj)<EOL><DEDENT>return value<EOL>

Loads a json value from a file and converts it to the corresponding python
    object.

    inputFilePath:
                    Path of the json file;

    Returns:
                    python value that represents the loaded json value


pass<EOL>

Write obj instance to Cap'n Proto object

.. warning: This is an abstract method.  Per abc protocol, attempts to
            subclass without overriding will fail.

:param proto: Cap'n Proto obj


<EOL>schema = self.getSchema()<EOL>proto = schema.new_message()<EOL>self.write(proto)<EOL>if packed:<EOL><INDENT>proto.write_packed(f)<EOL><DEDENT>else:<EOL><INDENT>proto.write(f)<EOL><DEDENT>

Write serialized object to file.

:param f: output file
:param packed: If true, will pack contents.


if len(pred) == <NUM_LIT:1>:<EOL><INDENT>return list(pred.keys())[<NUM_LIT:0>]<EOL><DEDENT>mostLikelyOutcome = None<EOL>maxProbability = <NUM_LIT:0><EOL>for prediction, probability in list(pred.items()):<EOL><INDENT>if probability > maxProbability:<EOL><INDENT>mostLikelyOutcome = prediction<EOL>maxProbability = probability<EOL><DEDENT><DEDENT>return mostLikelyOutcome<EOL>

Helper function to return a scalar value representing the most
            likely outcome given a probability distribution


infType, _, _= label.partition(cls._LABEL_SEPARATOR)<EOL>if not InferenceType.validate(infType):<EOL><INDENT>return None<EOL><DEDENT>return infType<EOL>

Extracts the PredictionKind (temporal vs. nontemporal) from the given
metric label.

:param label: (string) for a metric spec generated by
       :meth:`getMetricLabel`

:returns: (:class:`~nupic.frameworks.opf.opf_utils.InferenceType`)


def addInstance(self, groundTruth, prediction, record = None, result = None):

self.value = self.avg(prediction)<EOL>


self._windowSize = windowSize<EOL>self._countDict = dict()<EOL>self._history = deque([])<EOL>

:param windowSize:             The number of values that are used to compute the
                        moving average


def accumulate(self, groundTruth, prediction, accumulatedError, historyBuffer, result = None):

<EOL>if self.disabled:<EOL><INDENT>return <NUM_LIT:0><EOL><DEDENT>if historyBuffer is not None:<EOL><INDENT>historyBuffer.append((groundTruth, prediction[<NUM_LIT:0>]))<EOL>if len(historyBuffer) > self.spec.params["<STR_LIT>"] :<EOL><INDENT>historyBuffer.popleft()<EOL><DEDENT><DEDENT>return <NUM_LIT:0><EOL>

Accumulate history of groundTruth and "prediction" values.

For this metric, groundTruth is the actual category and "prediction" is a
dict containing one top-level item with a key of 0 (meaning this is the
0-step classificaton) and a value which is another dict, which contains the
probability for each category as output from the classifier. For example,
this is what "prediction" would be if the classifier said that category 0
had a 0.6 probability and category 1 had a 0.4 probability: {0:0.6, 1: 0.4}


<EOL>self.id = None<EOL>self.verbosity = <NUM_LIT:0><EOL>self.window = -<NUM_LIT:1><EOL>self.history = None<EOL>self.accumulatedError = <NUM_LIT:0><EOL>self.aggregateError = None<EOL>self.steps = <NUM_LIT:0><EOL>self.spec = metricSpec<EOL>self.disabled = False<EOL>self._predictionSteps = [<NUM_LIT:0>]<EOL>self._groundTruthHistory = deque([])<EOL>self._subErrorMetrics = None<EOL>self._maxRecords = None<EOL>if metricSpec is not None and metricSpec.params is not None:<EOL><INDENT>self.id = metricSpec.params.get('<STR_LIT:id>', None)<EOL>self._predictionSteps = metricSpec.params.get('<STR_LIT>', [<NUM_LIT:0>])<EOL>if not hasattr(self._predictionSteps, '<STR_LIT>'):<EOL><INDENT>self._predictionSteps = [self._predictionSteps]<EOL><DEDENT>self.verbosity = metricSpec.params.get('<STR_LIT>', <NUM_LIT:0>)<EOL>self._maxRecords = metricSpec.params.get('<STR_LIT>', None)<EOL>if '<STR_LIT>' in metricSpec.params:<EOL><INDENT>assert metricSpec.params['<STR_LIT>'] >= <NUM_LIT:1><EOL>self.history = deque([])<EOL>self.window = metricSpec.params['<STR_LIT>']<EOL><DEDENT>if '<STR_LIT>' in metricSpec.params:<EOL><INDENT>self._subErrorMetrics = []<EOL>for step in self._predictionSteps:<EOL><INDENT>subSpec = copy.deepcopy(metricSpec)<EOL>subSpec.params.pop('<STR_LIT>', None)<EOL>subSpec.params.pop('<STR_LIT>')<EOL>subSpec.metric = metricSpec.params['<STR_LIT>']<EOL>self._subErrorMetrics.append(getModule(subSpec))<EOL><DEDENT><DEDENT><DEDENT>

Initialize this metric

        If the params contains the key 'errorMetric', then that is the name of
        another metric to which we will pass a modified groundTruth and prediction
        to from our addInstance() method. For example, we may compute a moving mean
        on the groundTruth and then pass that to the AbsoluteAveError metric


``stats`` is expected to contain further information relevant to the given
metric, for example the number of timesteps represented in the current
measurement. All stats are implementation defined, and ``stats`` can be
``None``.

:returns: (dict) representing data from the metric
   ::

       {value : <current measurement>, "stats" : {<stat> : <value> ...}}


def aggregate(self, accumulatedError, historyBuffer, steps):

Updates the final aggregated score error given the prediction and the ground
truth.

:param accumulatedError: The total accumulated score from the previous
       predictions (possibly over some finite window)

:param historyBuffer: A buffer of the last <self.window> ground truth values
       that have been observed. If ``historyBuffer`` = None,  it means that
       no history is being kept.

:param steps: (int) The total number of (groundTruth, prediction) pairs that
       have been passed to the metric. This does not include pairs where
       ``groundTruth = SENTINEL_VALUE_FOR_MISSING_DATA``

:returns: The new aggregate (final) error measure.


def accumulate(self, groundTruth, prediction, accumulatedError, historyBuffer, result):

Updates the accumulated error given the prediction and the
ground truth.

:param groundTruth: Actual value that is observed for the current timestep

:param prediction: Value predicted by the network for the given timestep

:param accumulatedError: The total accumulated score from the previous
       predictions (possibly over some finite window)

:param historyBuffer: A buffer of the last <self.window> ground truth values
       that have been observed.

       If historyBuffer = None,  it means that no history is being kept.

:param result: An ModelResult class (see opf_utils.py), used for advanced
       metric calculation (e.g., MetricNegativeLogLikelihood)

:returns: The new accumulated error. That is:

    .. code-block:: python

       self.accumulatedError = self.accumulate(
         groundTruth, predictions, accumulatedError
       )

    ``historyBuffer`` should also be updated in this method.
    ``self.spec.params["window"]`` indicates the maximum size of the window.


return {"<STR_LIT:value>": self.value}<EOL>


@abstractmethod<EOL><INDENT>def addInstance(self, groundTruth, prediction, record = None, result = None):<DEDENT>

Add one instance consisting of ground truth and a prediction.

:param groundTruth:
  The actual measured value at the current timestep

:param prediction:
  The value predicted by the network at the current timestep

:param record: the raw input record as fed to
       :meth:`~nupic.frameworks.opf.model.Model.run` by the user. The
       typical usage is to feed a record to that method and get a
       :class:`~nupic.frameworks.opf.opf_utils.ModelResult`. Then you pass
       :class:`~nupic.frameworks.opf.opf_utils.ModelResult`.rawInput into
       this function as the record parameter.

:param result: (:class:`~nupic.frameworks.opf.opf_utils.ModelResult`) the
       result of running a row of data through an OPF model

:returns:
    The average error as computed over the metric's window size


assert isinstance(htmPredictionModel, HTMPredictionModel)<EOL>htmPredictionModel._getSPRegion().setParameter('<STR_LIT>', False)<EOL>return<EOL>

Disables learning in the HTMPredictionModel's Spatial Pooler, while
    retaining the ability to re-enable SP learning in the future.

    See also: htmPredictionModelControlEnableSPLearningCb.
    See also: model_callbacks.modelControlFinishLearningCb.

    htmPredictionModel:  pointer to a HTMPredictionModel instance

    Returns: nothing


self.__filePath = filePath<EOL>return<EOL>

filePath: path of file where TM __init__ args are to be saved


assert isinstance(htmPredictionModel, HTMPredictionModel)<EOL>htmPredictionModel._getTPRegion().setParameter('<STR_LIT>', False)<EOL>return<EOL>

Disables learning in the HTMPredictionModel's Temporal Pooler, while
    retaining the ability to re-enable TM learning in the future.

    See also: htmPredictionModelControlEnableTPLearningCb.
    See also: model_callbacks.modelControlFinishLearningCb.

    htmPredictionModel:  pointer to a HTMPredictionModel instance

    Returns: nothing


lastSegment = os.path.split(checkpointDir)[<NUM_LIT:1>]<EOL>if lastSegment[<NUM_LIT:0>] == '<STR_LIT:.>':<EOL><INDENT>return False<EOL><DEDENT>if not checkpointDir.endswith(g_defaultCheckpointExtension):<EOL><INDENT>return False<EOL><DEDENT>if not os.path.isdir(checkpointDir):<EOL><INDENT>return False<EOL><DEDENT>return True<EOL>

Return true iff checkpointDir appears to be a checkpoint directory.


json_helpers.validate(options.privateOptions,<EOL>schemaDict=g_parsedPrivateCommandLineOptionsSchema)<EOL>experimentDir = options.experimentDir<EOL>descriptionPyModule = helpers.loadExperimentDescriptionScriptFromDir(<EOL>experimentDir)<EOL>expIface = helpers.getExperimentDescriptionInterfaceFromModule(<EOL>descriptionPyModule)<EOL>if options.privateOptions['<STR_LIT>']:<EOL><INDENT>_printAvailableCheckpoints(experimentDir)<EOL>return None<EOL><DEDENT>experimentTasks = expIface.getModelControl().get('<STR_LIT>', [])<EOL>if (len(experimentTasks) == <NUM_LIT:0> and<EOL>expIface.getModelControl()['<STR_LIT>'] == OpfEnvironment.Nupic):<EOL><INDENT>expIface.convertNupicEnvToOPF()<EOL>experimentTasks = expIface.getModelControl().get('<STR_LIT>', [])<EOL><DEDENT>expIface.normalizeStreamSources()<EOL>newSerialization = options.privateOptions['<STR_LIT>']<EOL>if options.privateOptions['<STR_LIT>']:<EOL><INDENT>print("<STR_LIT>")<EOL>for label in [t['<STR_LIT>'] for t in experimentTasks]:<EOL><INDENT>print("<STR_LIT:\t>", label)<EOL><DEDENT>return None<EOL><DEDENT>if options.privateOptions['<STR_LIT>']:<EOL><INDENT>assert model is None<EOL>checkpointName = options.privateOptions['<STR_LIT>']<EOL>model = ModelFactory.loadFromCheckpoint(<EOL>savedModelDir=_getModelCheckpointDir(experimentDir, checkpointName),<EOL>newSerialization=newSerialization)<EOL><DEDENT>elif model is not None:<EOL><INDENT>print("<STR_LIT>")<EOL><DEDENT>else:<EOL><INDENT>modelDescription = expIface.getModelDescription()<EOL>model = ModelFactory.create(modelDescription)<EOL><DEDENT>if options.privateOptions['<STR_LIT>']:<EOL><INDENT>checkpointName = options.privateOptions['<STR_LIT>']<EOL>_saveModel(model=model,<EOL>experimentDir=experimentDir,<EOL>checkpointLabel=checkpointName,<EOL>newSerialization=newSerialization)<EOL>return model<EOL><DEDENT>taskIndexList = list(range(len(experimentTasks)))<EOL>customTaskExecutionLabelsList = options.privateOptions['<STR_LIT>']<EOL>if customTaskExecutionLabelsList:<EOL><INDENT>taskLabelsList = [t['<STR_LIT>'] for t in experimentTasks]<EOL>taskLabelsSet = set(taskLabelsList)<EOL>customTaskExecutionLabelsSet = set(customTaskExecutionLabelsList)<EOL>assert customTaskExecutionLabelsSet.issubset(taskLabelsSet),("<STR_LIT>"<EOL>"<STR_LIT>"<EOL>"<STR_LIT>") % (customTaskExecutionLabelsSet - taskLabelsSet,<EOL>customTaskExecutionLabelsList)<EOL>taskIndexList = [taskLabelsList.index(label) for label in<EOL>customTaskExecutionLabelsList]<EOL>print("<STR_LIT>" % [taskLabelsList[i] for<EOL>i in taskIndexList])<EOL><DEDENT>for taskIndex in taskIndexList:<EOL><INDENT>task = experimentTasks[taskIndex]<EOL>taskRunner = _TaskRunner(model=model,<EOL>task=task,<EOL>cmdOptions=options)<EOL>taskRunner.run()<EOL>del taskRunner<EOL>if options.privateOptions['<STR_LIT>']:<EOL><INDENT>_saveModel(model=model,<EOL>experimentDir=experimentDir,<EOL>checkpointLabel=task['<STR_LIT>'],<EOL>newSerialization=newSerialization)<EOL><DEDENT><DEDENT>return model<EOL>

Creates and runs the experiment

    Args:
      options: namedtuple ParseCommandLineOptionsResult
      model: For testing: may pass in an existing OPF Model instance
          to use instead of creating a new one.

    Returns: reference to OPFExperiment instance that was constructed (this
        is provided to aid with debugging) or None, if none was
        created.


self.__logger.debug("<STR_LIT>", self.__task['<STR_LIT>'])<EOL>if self.__cmdOptions.privateOptions['<STR_LIT>']:<EOL><INDENT>numIters = <NUM_LIT:10><EOL><DEDENT>else:<EOL><INDENT>numIters = self.__task['<STR_LIT>']<EOL><DEDENT>if numIters >= <NUM_LIT:0>:<EOL><INDENT>iterTracker = iter(range(numIters))<EOL><DEDENT>else:<EOL><INDENT>iterTracker = iter(itertools.count())<EOL><DEDENT>periodic = PeriodicActivityMgr(<EOL>requestedActivities=self._createPeriodicActivities())<EOL>self.__model.resetSequenceStates()<EOL>self.__taskDriver.setup()<EOL>while True:<EOL><INDENT>try:<EOL><INDENT>next(iterTracker)<EOL><DEDENT>except StopIteration:<EOL><INDENT>break<EOL><DEDENT>try:<EOL><INDENT>inputRecord = next(self.__datasetReader)<EOL><DEDENT>except StopIteration:<EOL><INDENT>break<EOL><DEDENT>result = self.__taskDriver.handleInputRecord(inputRecord=inputRecord)<EOL>if InferenceElement.encodings in result.inferences:<EOL><INDENT>result.inferences.pop(InferenceElement.encodings)<EOL><DEDENT>self.__predictionLogger.writeRecord(result)<EOL>periodic.tick()<EOL><DEDENT>self._getAndEmitExperimentMetrics(final=True)<EOL>self.__taskDriver.finalize()<EOL>self.__model.resetSequenceStates()<EOL>


print(parser.get_usage())<EOL>print(message)<EOL>sys.exit(<NUM_LIT:1>)<EOL>

Report usage error and exit program with error indication.


<EOL>for act in self.__activities:<EOL><INDENT>if not act.iteratorHolder[<NUM_LIT:0>]:<EOL><INDENT>continue<EOL><DEDENT>try:<EOL><INDENT>next(act.iteratorHolder[<NUM_LIT:0>])<EOL><DEDENT>except StopIteration:<EOL><INDENT>act.cb()<EOL>if act.repeating:<EOL><INDENT>act.iteratorHolder[<NUM_LIT:0>] = iter(range(act.period-<NUM_LIT:1>))<EOL><DEDENT>else:<EOL><INDENT>act.iteratorHolder[<NUM_LIT:0>] = None<EOL><DEDENT><DEDENT><DEDENT>return True<EOL>

Activity tick handler; services all activities

        Returns:
          True if controlling iterator says it's okay to keep going;
          False to stop


Place the model in a permanent "finished learning" mode.
        In such a mode the model will not be able to learn from subsequent input
        records.

        .. note:: Upon completion of this command, learning may not be resumed on
           the given instance of the model (e.g., the implementation may optimize
           itself by pruning data structures that are necessary for learning).


def __init__(self, inferenceType=None, proto=None):

assert inferenceType is not None and proto is None or (<EOL>inferenceType is None and proto is not None), (<EOL>"<STR_LIT>")<EOL>if proto is None:<EOL><INDENT>self._numPredictions = <NUM_LIT:0><EOL>self.__inferenceType =  inferenceType<EOL>self.__learningEnabled = True<EOL>self.__inferenceEnabled = True<EOL>self.__inferenceArgs = {}<EOL><DEDENT>else:<EOL><INDENT>self._numPredictions = proto.numPredictions<EOL>inferenceType = str(proto.inferenceType)<EOL>inferenceType = inferenceType[:<NUM_LIT:1>].upper() + inferenceType[<NUM_LIT:1>:]<EOL>self.__inferenceType = InferenceType.getValue(inferenceType)<EOL>self.__learningEnabled = proto.learningEnabled<EOL>self.__inferenceEnabled = proto.inferenceEnabled<EOL>self.__inferenceArgs = json.loads(proto.inferenceArgs)<EOL><DEDENT>

:param opf_utils.InferenceType inferenceType: mutually-exclusive with proto
                                              arg
:param proto: capnp ModelProto message reader for deserializing;
              mutually-exclusive with the other constructor args.


path = os.path.join(checkpointDir, "<STR_LIT>")<EOL>path = os.path.abspath(path)<EOL>return path<EOL>

Return the absolute path of the model's checkpoint file.

        :param checkpointDir: (string)
               Directory of where the experiment is to be or was saved
        :returns: (string) An absolute path.


<EOL>predictionNumber = self._numPredictions<EOL>self._numPredictions += <NUM_LIT:1><EOL>result = opf_utils.ModelResult(predictionNumber=predictionNumber,<EOL>rawInput=inputRecord)<EOL>return result<EOL>

Run one iteration of this model.

:param inputRecord: (object)
       A record object formatted according to
       :meth:`~nupic.data.record_stream.RecordStreamIface.getNextRecord` or
       :meth:`~nupic.data.record_stream.RecordStreamIface.getNextRecordDict`
       result format.
:returns: (:class:`~nupic.frameworks.opf.opf_utils.ModelResult`)
         An ModelResult namedtuple. The contents of ModelResult.inferences
         depends on the the specific inference type of this model, which
         can be queried by :meth:`.getInferenceType`.


path = os.path.join(saveModelDir, "<STR_LIT>")<EOL>path = os.path.abspath(path)<EOL>return path<EOL>

Return the absolute path to the directory where the model's own
        "extra data" are stored (i.e., data that's too big for pickling).

        :param saveModelDir: (string)
               Directory of where the experiment is to be or was saved
        :returns: (string) An absolute path.


logger = opf_utils.initLogger(cls)<EOL>logger.debug("<STR_LIT>", savedModelDir)<EOL>modelPickleFilePath = Model._getModelPickleFilePath(savedModelDir)<EOL>with open(modelPickleFilePath, '<STR_LIT:rb>') as modelPickleFile:<EOL><INDENT>logger.debug("<STR_LIT>")<EOL>model = pickle.load(modelPickleFile)<EOL>logger.debug("<STR_LIT>")<EOL><DEDENT>model._deSerializeExtraData(<EOL>extraDataDir=Model._getModelExtraDataDir(savedModelDir))<EOL>logger.debug("<STR_LIT>")<EOL>return model<EOL>

Load saved model.

        :param savedModelDir: (string)
               Directory of where the experiment is to be or was saved
        :returns: (:class:`Model`) The loaded model instance


Return the absolute path of the model's pickle file.

        :param saveModelDir: (string)
               Directory of where the experiment is to be or was saved
        :returns: (string) An absolute path.


raise NotImplementedError()<EOL>

Write state to proto object.

        The type of proto is determined by :meth:`getSchema`.


checkpointPath = cls._getModelCheckpointFilePath(checkpointDir)<EOL>with open(checkpointPath, '<STR_LIT:r>') as f:<EOL><INDENT>proto = cls.getSchema().read(f,<EOL>traversal_limit_in_words=_TRAVERSAL_LIMIT_IN_WORDS)<EOL><DEDENT>model = cls.read(proto)<EOL>return model<EOL>

Deserializes model from checkpointDir using capnproto


Get runtime statistics specific to this model, i.e.
        ``activeCellOverlapAvg``.

        :returns: (dict) A {statistic names: stats} dictionary


return self.__inferenceType<EOL>

Return the InferenceType of this model.
        This is immutable.

        :returns: :class:`~nupic.frameworks.opf.opf_utils.InferenceType`


return self.__inferenceArgs<EOL>

Return the dict of arguments for the current inference mode.

        :returns: (dict) The arguments of the inference mode


self.__learningEnabled = False<EOL>return<EOL>

Turn Learning off for the current model.


return self.__logger<EOL>

Get the logger for this object. This is a protected method that is used
        by the Model to access the logger created by the subclass

        return:
          A logging.Logger object. Should not be None


self.__dict__.update(state)<EOL>self.__restoringFromState = True<EOL>self.__logger = initLogger(self)<EOL>if not hasattr(self, "<STR_LIT>"):<EOL><INDENT>self.__restoringFromV1 = True<EOL>self._hasSP = True<EOL>if self.__temporalNetInfo is not None:<EOL><INDENT>self._Model__inferenceType = InferenceType.TemporalNextStep<EOL>self._netInfo = self.__temporalNetInfo<EOL>self._hasTP = True<EOL><DEDENT>else:<EOL><INDENT>raise RuntimeError("<STR_LIT>")<EOL><DEDENT>self._Model__inferenceArgs = {}<EOL>self._Model__learningEnabled = True<EOL>self._Model__inferenceEnabled = True<EOL>self.__dict__.pop("<STR_LIT>", None)<EOL>self.__dict__.pop("<STR_LIT>", None)<EOL>self.__dict__.pop("<STR_LIT>", None)<EOL><DEDENT>if not hasattr(self, "<STR_LIT>"):<EOL><INDENT>self._hasSP = False<EOL>self._hasTP = False<EOL>if self.__encoderNetInfo is not None:<EOL><INDENT>self._netInfo = self.__encoderNetInfo<EOL><DEDENT>elif self.__nonTemporalNetInfo is not None:<EOL><INDENT>self._netInfo = self.__nonTemporalNetInfo<EOL>self._hasSP = True<EOL><DEDENT>else:<EOL><INDENT>self._netInfo = self.__temporalNetInfo<EOL>self._hasSP = True<EOL>self._hasTP = True<EOL><DEDENT>self.__dict__.pop("<STR_LIT>", None)<EOL>self.__dict__.pop("<STR_LIT>", None)<EOL>self.__dict__.pop("<STR_LIT>", None)<EOL><DEDENT>self._classifierInputEncoder = None<EOL>if not hasattr(self, '<STR_LIT>'):<EOL><INDENT>self._minLikelihoodThreshold = DEFAULT_LIKELIHOOD_THRESHOLD<EOL><DEDENT>if not hasattr(self, '<STR_LIT>'):<EOL><INDENT>self._maxPredictionsPerStep = DEFAULT_MAX_PREDICTIONS_PER_STEP<EOL><DEDENT>if not hasattr(self, '<STR_LIT>'):<EOL><INDENT>self._hasCL = (self._getClassifierRegion() is not None)<EOL><DEDENT>self.__logger.debug("<STR_LIT>" % self.__class__.__name__)<EOL>

Set the state of ourself from a serialized state.

See also: _deSerializeExtraData


inferenceArgs = self.getInferenceArgs()<EOL>predictedFieldName = inferenceArgs.get('<STR_LIT>', None)<EOL>if predictedFieldName is None:<EOL><INDENT>raise ValueError(<EOL>"<STR_LIT>"<EOL>)<EOL><DEDENT>self._predictedFieldName = predictedFieldName<EOL>classifier = self._getClassifierRegion()<EOL>if not self._hasCL or classifier is None:<EOL><INDENT>return {}<EOL><DEDENT>sensor = self._getSensorRegion()<EOL>minLikelihoodThreshold = self._minLikelihoodThreshold<EOL>maxPredictionsPerStep = self._maxPredictionsPerStep<EOL>needLearning = self.isLearningEnabled()<EOL>inferences = {}<EOL>if self._classifierInputEncoder is None:<EOL><INDENT>if predictedFieldName is None:<EOL><INDENT>raise RuntimeError("<STR_LIT>"<EOL>"<STR_LIT>"<EOL>"<STR_LIT>")<EOL><DEDENT>encoderList = sensor.getSelf().encoder.getEncoderList()<EOL>self._numFields = len(encoderList)<EOL>fieldNames = sensor.getSelf().encoder.getScalarNames()<EOL>if predictedFieldName in fieldNames:<EOL><INDENT>self._predictedFieldIdx = fieldNames.index(predictedFieldName)<EOL><DEDENT>else:<EOL><INDENT>self._predictedFieldIdx = None<EOL><DEDENT>if sensor.getSelf().disabledEncoder is not None:<EOL><INDENT>encoderList = sensor.getSelf().disabledEncoder.getEncoderList()<EOL><DEDENT>else:<EOL><INDENT>encoderList = []<EOL><DEDENT>if len(encoderList) >= <NUM_LIT:1>:<EOL><INDENT>fieldNames = sensor.getSelf().disabledEncoder.getScalarNames()<EOL>self._classifierInputEncoder = encoderList[fieldNames.index(<EOL>predictedFieldName)]<EOL><DEDENT>else:<EOL><INDENT>encoderList = sensor.getSelf().encoder.getEncoderList()<EOL>self._classifierInputEncoder = encoderList[self._predictedFieldIdx]<EOL><DEDENT><DEDENT>if not predictedFieldName in rawInput:<EOL><INDENT>raise ValueError("<STR_LIT>"<EOL>"<STR_LIT>"<EOL>% predictedFieldName)<EOL><DEDENT>absoluteValue = rawInput[predictedFieldName]<EOL>bucketIdx = self._classifierInputEncoder.getBucketIndices(absoluteValue)[<NUM_LIT:0>]<EOL>if isinstance(self._classifierInputEncoder, DeltaEncoder):<EOL><INDENT>if not hasattr(self,"<STR_LIT>"):<EOL><INDENT>self._ms_prevVal = absoluteValue<EOL><DEDENT>prevValue = self._ms_prevVal<EOL>self._ms_prevVal = absoluteValue<EOL>actualValue = absoluteValue - prevValue<EOL><DEDENT>else:<EOL><INDENT>actualValue = absoluteValue<EOL><DEDENT>if isinstance(actualValue, float) and math.isnan(actualValue):<EOL><INDENT>actualValue = SENTINEL_VALUE_FOR_MISSING_DATA<EOL><DEDENT>classifier.setParameter('<STR_LIT>', True)<EOL>classifier.setParameter('<STR_LIT>', needLearning)<EOL>classificationIn = {'<STR_LIT>': bucketIdx,<EOL>'<STR_LIT>': actualValue}<EOL>if inputTSRecordIdx is not None:<EOL><INDENT>recordNum = inputTSRecordIdx<EOL><DEDENT>else:<EOL><INDENT>recordNum = self.__numRunCalls<EOL><DEDENT>clResults = classifier.getSelf().customCompute(recordNum=recordNum,<EOL>patternNZ=patternNZ,<EOL>classification=classificationIn)<EOL>predictionSteps = classifier.getParameter('<STR_LIT>')<EOL>predictionSteps = [int(x) for x in predictionSteps.split('<STR_LIT:U+002C>')]<EOL>inferences[InferenceElement.multiStepPredictions] = dict()<EOL>inferences[InferenceElement.multiStepBestPredictions] = dict()<EOL>inferences[InferenceElement.multiStepBucketLikelihoods] = dict()<EOL>for steps in predictionSteps:<EOL><INDENT>likelihoodsVec = clResults[steps]<EOL>bucketValues = clResults['<STR_LIT>']<EOL>likelihoodsDict = dict()<EOL>bestActValue = None<EOL>bestProb = None<EOL>for (actValue, prob) in zip(bucketValues, likelihoodsVec):<EOL><INDENT>if actValue in likelihoodsDict:<EOL><INDENT>likelihoodsDict[actValue] += prob<EOL><DEDENT>else:<EOL><INDENT>likelihoodsDict[actValue] = prob<EOL><DEDENT>if bestProb is None or likelihoodsDict[actValue] > bestProb:<EOL><INDENT>bestProb = likelihoodsDict[actValue]<EOL>bestActValue = actValue<EOL><DEDENT><DEDENT>likelihoodsDict = HTMPredictionModel._removeUnlikelyPredictions(<EOL>likelihoodsDict, minLikelihoodThreshold, maxPredictionsPerStep)<EOL>bucketLikelihood = {}<EOL>for k in likelihoodsDict.keys():<EOL><INDENT>bucketLikelihood[self._classifierInputEncoder.getBucketIndices(k)[<NUM_LIT:0>]] = (<EOL>likelihoodsDict[k])<EOL><DEDENT>if isinstance(self._classifierInputEncoder, DeltaEncoder):<EOL><INDENT>if not hasattr(self, '<STR_LIT>'):<EOL><INDENT>self._ms_predHistories = dict()<EOL><DEDENT>predHistories = self._ms_predHistories<EOL>if not steps in predHistories:<EOL><INDENT>predHistories[steps] = deque()<EOL><DEDENT>predHistory = predHistories[steps]<EOL>sumDelta = sum(predHistory)<EOL>offsetDict = dict()<EOL>for (k, v) in likelihoodsDict.iteritems():<EOL><INDENT>if k is not None:<EOL><INDENT>offsetDict[absoluteValue+float(k)+sumDelta] = v<EOL><DEDENT><DEDENT>bucketLikelihoodOffset = {}<EOL>for k in offsetDict.keys():<EOL><INDENT>bucketLikelihoodOffset[self._classifierInputEncoder.getBucketIndices(k)[<NUM_LIT:0>]] = (<EOL>offsetDict[k])<EOL><DEDENT>if bestActValue is not None:<EOL><INDENT>predHistory.append(bestActValue)<EOL><DEDENT>if len(predHistory) >= steps:<EOL><INDENT>predHistory.popleft()<EOL><DEDENT>if len(offsetDict)><NUM_LIT:0>:<EOL><INDENT>inferences[InferenceElement.multiStepPredictions][steps] = offsetDict<EOL>inferences[InferenceElement.multiStepBucketLikelihoods][steps] = bucketLikelihoodOffset<EOL><DEDENT>else:<EOL><INDENT>inferences[InferenceElement.multiStepPredictions][steps] = likelihoodsDict<EOL>inferences[InferenceElement.multiStepBucketLikelihoods][steps] = bucketLikelihood<EOL><DEDENT>if bestActValue is None:<EOL><INDENT>inferences[InferenceElement.multiStepBestPredictions][steps] = None<EOL><DEDENT>else:<EOL><INDENT>inferences[InferenceElement.multiStepBestPredictions][steps] = (<EOL>absoluteValue + sumDelta + bestActValue)<EOL><DEDENT><DEDENT>else:<EOL><INDENT>inferences[InferenceElement.multiStepPredictions][steps] = (<EOL>likelihoodsDict)<EOL>inferences[InferenceElement.multiStepBestPredictions][steps] = (<EOL>bestActValue)<EOL>inferences[InferenceElement.multiStepBucketLikelihoods][steps] = (<EOL>bucketLikelihood)<EOL><DEDENT><DEDENT>return inferences<EOL>

Handle the CLA Classifier compute logic when implementing multi-step
        prediction. This is where the patternNZ is associated with one of the
        other fields from the dataset 0 to N steps in the future. This method is
        used by each type of network (encoder only, SP only, SP +TM) to handle the
        compute logic through the CLA Classifier. It fills in the inference dict with
        the results of the compute.

        Parameters:
        -------------------------------------------------------------------
        patternNZ: The input to the CLA Classifier as a list of active input indices
        inputTSRecordIdx: The index of the record as computed from the timestamp
                      and aggregation interval. This normally increments by 1
                      each time unless there are missing records. If there is no
                      aggregation interval or timestamp in the data, this will be
                      None.
        rawInput:   The raw input to the sensor, as a dict.


return self._getSensorRegion().getSelf().dataSource<EOL>

Returns: data source that we installed in sensor region


def __init__(self, net, statsCollectors):

self.net = net<EOL>self.statsCollectors = statsCollectors<EOL>return<EOL>

net:          The CLA Network instance
statsCollectors:
              Sequence of 0 or more CLAStatistic-based instances


return  self._getSensorRegion().getSelf().disabledEncoder<EOL>

Returns:  sensor region's encoder that is sent only to the classifier,
            not to the bottom of the network


def __createHTMNetwork(self, sensorParams, spEnable, spParams, tmEnable,<EOL>tmParams, clEnable, clParams, anomalyParams):

<EOL>n = Network()<EOL>n.addRegion("<STR_LIT>", "<STR_LIT>", json.dumps(dict(verbosity=sensorParams['<STR_LIT>'])))<EOL>sensor = n.regions['<STR_LIT>'].getSelf()<EOL>enabledEncoders = copy.deepcopy(sensorParams['<STR_LIT>'])<EOL>for name, params in enabledEncoders.items():<EOL><INDENT>if params is not None:<EOL><INDENT>classifierOnly = params.pop('<STR_LIT>', False)<EOL>if classifierOnly:<EOL><INDENT>enabledEncoders.pop(name)<EOL><DEDENT><DEDENT><DEDENT>disabledEncoders = copy.deepcopy(sensorParams['<STR_LIT>'])<EOL>for name, params in disabledEncoders.items():<EOL><INDENT>if params is None:<EOL><INDENT>disabledEncoders.pop(name)<EOL><DEDENT>else:<EOL><INDENT>classifierOnly = params.pop('<STR_LIT>', False)<EOL>if not classifierOnly:<EOL><INDENT>disabledEncoders.pop(name)<EOL><DEDENT><DEDENT><DEDENT>encoder = MultiEncoder(enabledEncoders)<EOL>sensor.encoder = encoder<EOL>sensor.disabledEncoder = MultiEncoder(disabledEncoders)<EOL>sensor.dataSource = DataBuffer()<EOL>prevRegion = "<STR_LIT>"<EOL>prevRegionWidth = encoder.getWidth()<EOL>if spEnable:<EOL><INDENT>spParams = spParams.copy()<EOL>spParams['<STR_LIT>'] = prevRegionWidth<EOL>self.__logger.debug("<STR_LIT>" % spParams)<EOL>n.addRegion("<STR_LIT>", "<STR_LIT>", json.dumps(spParams))<EOL>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>")<EOL>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL>prevRegion = "<STR_LIT>"<EOL>prevRegionWidth = spParams['<STR_LIT>']<EOL><DEDENT>if tmEnable:<EOL><INDENT>tmParams = tmParams.copy()<EOL>if prevRegion == '<STR_LIT>':<EOL><INDENT>tmParams['<STR_LIT>'] = tmParams['<STR_LIT>'] = prevRegionWidth<EOL><DEDENT>else:<EOL><INDENT>assert tmParams['<STR_LIT>'] == prevRegionWidth<EOL>tmParams['<STR_LIT>'] = tmParams['<STR_LIT>']<EOL><DEDENT>self.__logger.debug("<STR_LIT>" % tmParams)<EOL>n.addRegion("<STR_LIT>", "<STR_LIT>", json.dumps(tmParams))<EOL>n.link(prevRegion, "<STR_LIT>", "<STR_LIT>", "<STR_LIT>")<EOL>if prevRegion != "<STR_LIT>":<EOL><INDENT>n.link("<STR_LIT>", prevRegion, "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL><DEDENT>else:<EOL><INDENT>n.link("<STR_LIT>", prevRegion, "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL><DEDENT>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL>prevRegion = "<STR_LIT>"<EOL>prevRegionWidth = tmParams['<STR_LIT>']<EOL><DEDENT>if clEnable and clParams is not None:<EOL><INDENT>clParams = clParams.copy()<EOL>clRegionName = clParams.pop('<STR_LIT>')<EOL>self.__logger.debug("<STR_LIT>" % (clRegionName,<EOL>clParams))<EOL>n.addRegion("<STR_LIT>", "<STR_LIT>" % str(clRegionName), json.dumps(clParams))<EOL>if str(clRegionName) == "<STR_LIT>":<EOL><INDENT>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL><DEDENT>n.link("<STR_LIT>", "<STR_LIT>", "<STR_LIT>", "<STR_LIT>", srcOutput="<STR_LIT>",<EOL>destInput="<STR_LIT>")<EOL>n.link(prevRegion, "<STR_LIT>", "<STR_LIT>", "<STR_LIT>")<EOL><DEDENT>if self.getInferenceType() == InferenceType.TemporalAnomaly:<EOL><INDENT>anomalyClParams = dict(<EOL>trainRecords=anomalyParams.get('<STR_LIT>', None),<EOL>cacheSize=anomalyParams.get('<STR_LIT>', None)<EOL>)<EOL>self._addAnomalyClassifierRegion(n, anomalyClParams, spEnable, tmEnable)<EOL><DEDENT>n.initialize()<EOL>return NetworkInfo(net=n, statsCollectors=[])<EOL>

Create a CLA network and return it.

        description:  HTMPredictionModel description dictionary (TODO: define schema)
        Returns:      NetworkInfo instance;


return self._netInfo.net.regions['<STR_LIT>']<EOL>

Returns reference to the network's Sensor region


if (self._netInfo.net is not None and<EOL>"<STR_LIT>" in self._netInfo.net.regions):<EOL><INDENT>return self._netInfo.net.regions["<STR_LIT>"]<EOL><DEDENT>else:<EOL><INDENT>return None<EOL><DEDENT>

Returns reference to the network's Classifier region


if not inferenceType in self.__supportedInferenceKindSet:<EOL><INDENT>raise ValueError("<STR_LIT>".format(self.__class__, inferenceType))<EOL><DEDENT>if baseProto is None:<EOL><INDENT>super(HTMPredictionModel, self).__init__(inferenceType)<EOL><DEDENT>else:<EOL><INDENT>super(HTMPredictionModel, self).__init__(proto=baseProto)<EOL><DEDENT>self.__restoringFromState = False<EOL>self.__restoringFromV1 = False<EOL>self.__logger = initLogger(self)<EOL>self.__logger.debug("<STR_LIT>" % self.__myClassName)<EOL>self._minLikelihoodThreshold = minLikelihoodThreshold<EOL>self._maxPredictionsPerStep = maxPredictionsPerStep<EOL>self.__spLearningEnabled = bool(spEnable)<EOL>self.__tpLearningEnabled = bool(tmEnable)<EOL>if not InferenceType.isTemporal(self.getInferenceType())or self.getInferenceType() == InferenceType.NontemporalMultiStep:<EOL><INDENT>tmEnable = False<EOL><DEDENT>self._netInfo = None<EOL>self._hasSP = spEnable<EOL>self._hasTP = tmEnable<EOL>self._hasCL = clEnable<EOL>self._classifierInputEncoder = None<EOL>self._predictedFieldIdx = None<EOL>self._predictedFieldName = None<EOL>self._numFields = None<EOL>if network is not None:<EOL><INDENT>self._netInfo = NetworkInfo(net=network, statsCollectors=[])<EOL><DEDENT>else:<EOL><INDENT>self._netInfo = self.__createHTMNetwork(<EOL>sensorParams, spEnable, spParams, tmEnable, tmParams, clEnable,<EOL>clParams, anomalyParams)<EOL><DEDENT>if self.getInferenceType() == InferenceType.NontemporalAnomaly:<EOL><INDENT>self._getSPRegion().setParameter("<STR_LIT>", True)<EOL><DEDENT>if self.getInferenceType() == InferenceType.TemporalAnomaly:<EOL><INDENT>self._getTPRegion().setParameter("<STR_LIT>", True)<EOL><DEDENT>self.__trainSPNetOnlyIfRequested = trainSPNetOnlyIfRequested<EOL>self.__numRunCalls = <NUM_LIT:0><EOL>self.__finishedLearning = False<EOL>self.__logger.debug("<STR_LIT>" % self.__class__.__name__)<EOL>self._input = None<EOL>return<EOL>

:param network: if not None, the deserialized nupic.engine.Network instance
                to use instead of creating a new Network
:param baseProto: if not None, capnp ModelProto message reader for
                  deserializing; supersedes inferenceType


ret = {"<STR_LIT>" : self.__numRunCalls}<EOL>temporalStats = dict()<EOL>if self._hasTP:<EOL><INDENT>for stat in self._netInfo.statsCollectors:<EOL><INDENT>sdict = stat.getStats()<EOL>temporalStats.update(sdict)<EOL><DEDENT><DEDENT>ret[InferenceType.getLabel(InferenceType.TemporalNextStep)] = temporalStats<EOL>return ret<EOL>

Only returns data for a stat called ``numRunCalls``.
:return:


obj = object.__new__(cls)<EOL>super(HTMPredictionModel, obj).__init__(proto=proto.modelBase)<EOL>obj._minLikelihoodThreshold = round(proto.minLikelihoodThreshold,<EOL>EPSILON_ROUND)<EOL>obj._maxPredictionsPerStep = proto.maxPredictionsPerStep<EOL>network = Network.read(proto.network)<EOL>obj._hasSP = ("<STR_LIT>" in network.regions)<EOL>obj._hasTP = ("<STR_LIT>" in network.regions)<EOL>obj._hasCL = ("<STR_LIT>" in network.regions)<EOL>obj._netInfo = NetworkInfo(net=network, statsCollectors=[])<EOL>obj.__spLearningEnabled = bool(proto.spLearningEnabled)<EOL>obj.__tpLearningEnabled = bool(proto.tpLearningEnabled)<EOL>obj.__numRunCalls = proto.numRunCalls<EOL>obj._classifierInputEncoder = None<EOL>if proto.predictedFieldIdx.which() == "<STR_LIT:none>":<EOL><INDENT>obj._predictedFieldIdx = None<EOL><DEDENT>else:<EOL><INDENT>obj._predictedFieldIdx = proto.predictedFieldIdx.value<EOL><DEDENT>if proto.predictedFieldName.which() == "<STR_LIT:none>":<EOL><INDENT>obj._predictedFieldName = None<EOL><DEDENT>else:<EOL><INDENT>obj._predictedFieldName = proto.predictedFieldName.value<EOL><DEDENT>obj._numFields = proto.numFields<EOL>if proto.numFields.which() == "<STR_LIT:none>":<EOL><INDENT>obj._numFields = None<EOL><DEDENT>else:<EOL><INDENT>obj._numFields = proto.numFields.value<EOL><DEDENT>obj.__trainSPNetOnlyIfRequested = proto.trainSPNetOnlyIfRequested<EOL>obj.__finishedLearning = proto.finishedLearning<EOL>obj._input = None<EOL>sensor = network.regions['<STR_LIT>'].getSelf()<EOL>sensor.dataSource = DataBuffer()<EOL>network.initialize()<EOL>obj.__logger = initLogger(obj)<EOL>obj.__logger.debug("<STR_LIT>" % obj.__myClassName)<EOL>obj.__restoringFromState = False<EOL>obj.__restoringFromV1 = False<EOL>return obj<EOL>

:param proto: capnp HTMPredictionModelProto message reader


assert self.__restoringFromState<EOL>assert (self._netInfo.net is None), "<STR_LIT>"<EOL>stateDir = self.__getNetworkStateDirectory(extraDataDir=extraDataDir)<EOL>self.__logger.debug(<EOL>"<STR_LIT>", self)<EOL>self._netInfo.net = Network(stateDir)<EOL>self.__logger.debug(<EOL>"<STR_LIT>", self)<EOL>self._netInfo.net.initialize()<EOL>if self.getInferenceType() == InferenceType.TemporalAnomaly:<EOL><INDENT>classifierType = self._getAnomalyClassifier().getSelf().__class__.__name__<EOL>if classifierType is '<STR_LIT>':<EOL><INDENT>anomalyClParams = dict(<EOL>trainRecords=self._classifier_helper._autoDetectWaitRecords,<EOL>cacheSize=self._classifier_helper._history_length,<EOL>)<EOL>spEnable = (self._getSPRegion() is not None)<EOL>tmEnable = True<EOL>knnRegion = self._getAnomalyClassifier().getSelf()<EOL>self._addAnomalyClassifierRegion(self._netInfo.net, anomalyClParams,<EOL>spEnable, tmEnable)<EOL>self._getAnomalyClassifier().getSelf()._iteration = self.__numRunCalls<EOL>self._getAnomalyClassifier().getSelf()._recordsCache = (<EOL>self._classifier_helper.saved_states)<EOL>self._getAnomalyClassifier().getSelf().saved_categories = (<EOL>self._classifier_helper.saved_categories)<EOL>self._getAnomalyClassifier().getSelf()._knnclassifier = knnRegion<EOL>self._getTPRegion().setParameter('<STR_LIT>', True)<EOL>del self._classifier_helper<EOL>self._netInfo.net.initialize()<EOL><DEDENT><DEDENT>self.__restoringFromState = False<EOL>self.__logger.debug("<STR_LIT>", self)<EOL>return<EOL>

[virtual method override] This method is called during deserialization
        (after __setstate__) with an external directory path that can be used to
        bypass pickle for loading large binary states.

        extraDataDir:
                      Model's extra data directory path


@requireAnomalyModel<EOL><INDENT>def setAnomalyParameter(self, param, value):<DEDENT>

self._getAnomalyClassifier().setParameter(param, value)<EOL>

Set a parameter of the anomaly classifier within this model.

:param param: (string) name of parameter to set
:param value: (object) value to set


def __init__(self, experimentDir, label, inferenceType,<EOL>fields, metricNames=None, checkpointSource=None):

<EOL>self.__experimentDir = experimentDir<EOL>self.__inferenceType = inferenceType<EOL>self.__inputFieldsMeta = tuple(copy.deepcopy(fields))<EOL>self.__numInputFields = len(self.__inputFieldsMeta)<EOL>self.__label = label<EOL>if metricNames is not None:<EOL><INDENT>metricNames.sort()<EOL><DEDENT>self.__metricNames = metricNames<EOL>self.__outputFieldsMeta = []<EOL>self._rawInputNames = []<EOL>self.__datasetPath = None<EOL>self.__dataset = None<EOL>self.__checkpointCache = None<EOL>if checkpointSource is not None:<EOL><INDENT>checkpointSource.seek(<NUM_LIT:0>)<EOL>self.__checkpointCache = io.StringIO()<EOL>shutil.copyfileobj(checkpointSource, self.__checkpointCache)<EOL><DEDENT>return<EOL>

Constructor

        experimentDir:
                      experiment directory path that contains description.py

        label:        A label string to incorporate into the filename.

        inferenceElements:

        inferenceType:
                      An constant from opf_utils.InferenceType for the
                      requested prediction writer

        fields:       a non-empty sequence of nupic.data.fieldmeta.FieldMetaInfo
                      representing fields that will be emitted to this prediction
                      writer

        metricNames:  OPTIONAL - A list of metric names that well be emiited by this
                      prediction writer

        checkpointSource:
                      If not None, a File-like object containing the
                      previously-checkpointed predictions for setting the initial
                      contents of this PredictionOutputStream.  Will be copied
                      before returning, if needed.


Closes the writer (e.g., close the underlying file)


def checkpoint(self, checkpointSink, maxRows):

checkpointSink.truncate()<EOL>if self.__dataset is None:<EOL><INDENT>if self.__checkpointCache is not None:<EOL><INDENT>self.__checkpointCache.seek(<NUM_LIT:0>)<EOL>shutil.copyfileobj(self.__checkpointCache, checkpointSink)<EOL>checkpointSink.flush()<EOL>return<EOL><DEDENT>else:<EOL><INDENT>return<EOL><DEDENT><DEDENT>self.__dataset.flush()<EOL>totalDataRows = self.__dataset.getDataRowCount()<EOL>if totalDataRows == <NUM_LIT:0>:<EOL><INDENT>return<EOL><DEDENT>reader = FileRecordStream(self.__datasetPath, missingValues=[])<EOL>writer = csv.writer(checkpointSink)<EOL>writer.writerow(reader.getFieldNames())<EOL>numToWrite = min(maxRows, totalDataRows)<EOL>numRowsToSkip = totalDataRows - numToWrite<EOL>for i in range(numRowsToSkip):<EOL><INDENT>next(reader)<EOL><DEDENT>numWritten = <NUM_LIT:0><EOL>while True:<EOL><INDENT>row = reader.getNextRecord()<EOL>if row is None:<EOL><INDENT>break;<EOL><DEDENT>row =  [str(element) for element in row]<EOL>writer.writerow(row)<EOL>numWritten +=<NUM_LIT:1><EOL><DEDENT>assert numWritten == numToWrite,"<STR_LIT>" % (numWritten, numToWrite)<EOL>checkpointSink.flush()<EOL>return<EOL>

[virtual method override] Save a checkpoint of the prediction output
        stream. The checkpoint comprises up to maxRows of the most recent inference
        records.

        Parameters:
        ----------------------------------------------------------------------
        checkpointSink:     A File-like object where predictions checkpoint data, if
                            any, will be stored.
        maxRows:            Maximum number of most recent inference rows
                            to checkpoint.


Emits periodic metrics to stdout in JSON.

        :param metrics: A list of metrics as returned by
              :meth:`nupic.frameworks.opf.opf_task_driver.OPFTaskDriver.getMetrics`.


:returns:     The next record from the dataset.  The returned record object
              is of the same structure as returned by
              :meth:`nupic.data.record_stream.RecordStreamIface.getNextRecord`.
              Returns ``None`` if the next record is not available yet.

:raises: (StopIteration) if a hard "end of file" has been reached
              and no more records will be forthcoming.


Emits final metrics.

        .. note:: the intention is that the final metrics may go to a different
                  place (e.g., csv file) versus :meth:`emitPeriodicMetrics`
                  (e.g., stdout)

        :param metrics: A list of metrics as returned by
              :meth:`nupic.frameworks.opf.opf_task_driver.OPFTaskDriver.getMetrics`.


if self.__dataset:<EOL><INDENT>self.__dataset.close()<EOL><DEDENT>self.__dataset = None<EOL>return<EOL>

[virtual method override] Closes the writer (e.g., close the underlying
        file)


@abstractmethod<EOL><INDENT>def writeRecords(self, modelResults, progressCB=None):<DEDENT>

Same as :meth:`writeRecord`, but emits multiple rows in one shot.

:param modelResults: (list) of
       :class:`nupic.frameworks.opf.opf_utils.ModelResult` objects, each
       represents one record.
:param progressCB: (func) optional callback method that will be called after
       each batch of records is written.