[WIP] Setting parameters via crossvalidation

Hi,

this is a work-in-progress PR that addresses #44  and partially #50. Progress report:
 - [x] refactored the training code so that it doesn't crash if a trained model is not present 
 - [x] modified trainer so that CRF parameters can be set
 - [x] added a function that calculates accuracy over crossvalidation. This is a bit naive in that I score per-token and pretend multi-token entities do not exits. The proper way to do it would be to use the CONLL NER script, but I feel that would be too much work. My colleague assures me the differences between the correct approach and my hack will not be large for the kind of data we have here. 
 - [ ] perform grid search over parameter values to find good parameter settings
 - [ ] add extra features to improve performance (e.g. suffix might be useful)
 - [ ] comment and update user manual
That makes sense.

On Sun, Oct 5, 2014 at 5:32 PM, Miroslav Batchkarov <
notifications@github.com> wrote:

> In training/utils.py:
>
> > +                # sometimes there are more/less gold standard labels than
> > +                # the number of tags the system will output, This is because
> > +                # our tokenizer works differently to how the data is tokenized.
> > +                # Let's pretend this never happened
> > +                # print 'WARNING: data tokenized incorrectly'
> > +                continue
> > +            predictions = usaddress.tag(address_text)
> > +
> > +            for pred_labels, pred_tokens in predictions.items():
> > +                for _ in usaddress.tokenize(pred_tokens):
> > +                    y_pred.append(pred_labels)
> > +            y_true.extend(list(labels))
> > +            assert len(y_pred) == len(y_true)
> > +
> > +        # print classification_report(y_true, y_pred)
> > +        score = f1_score(y_true, y_pred)
>
> Currently it's the latter- each predicted label for a token matches the
> gold standard.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/datamade/us-address-parser/pull/53/files#r18438138>.
>



-- 
773.888.2718