Repository: nyu-mll/jiant

Issue:
Evaluating more than once on the same task
> 07/23 06:00:44 PM: Config overrides: global_ro_exp_dir = /misc/vlgscratch4/BowmanGroup/sbowman/exp/default, project_dir = /misc/vlgscratch4/BowmanGroup/sbowman/exp/, do_train = 0, train_for_eval = 0, ev
> al_tasks = "mnli,mnli-diagnostic", reload_tasks = 1, reload_indexing = 1, reindex_tasks = "mnli-diagnostic,mnli", exp_name = mnli-random-fifth, train_tasks = mnli, training_data_fraction = 0.2, run_name
>  = elmo-do2-sd2, elmo_chars_only = 0, dropout = 0.2

...
> 07/23 06:09:05 PM: Loaded model state from /misc/vlgscratch4/BowmanGroup/sbowman/exp//mnli-random-fifth/elmo-do2-sd2/model_state_eval_best.th
> 07/23 06:09:05 PM: Evaluating...
> 07/23 06:09:05 PM: Evaluating on: mnli
> 07/23 06:14:29 PM: Task 'mnli': sorting predictions by 'idx'
> 07/23 06:14:29 PM: Evaluating on: mnli
> 07/23 06:19:54 PM: Task 'mnli': sorting predictions by 'idx'
> 07/23 06:19:54 PM: Evaluating on: mnli-diagnostic
> /home/sbowman/miniconda3/envs/jiant/lib/python3.6/site-packages/sklearn/metrics/classification.py:538: RuntimeWarning: invalid value encountered in double_scalars
>   mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)
> 07/23 06:20:15 PM: Task 'mnli-diagnostic': sorting predictions by 'idx'
> 07/23 06:20:15 PM: Evaluating on: mnli
> 07/23 06:25:59 PM: Task 'mnli': sorting predictions by 'idx'
> 07/23 06:25:59 PM: Evaluating on: mnli
...


Discussion:
May involve #191
One more clue:

> 07/23 06:08:29 PM:      Finished loading tasks in 464.917s
> 07/23 06:08:29 PM:       Tasks: ['mnli', 'mnli', 'mnli-diagnostic']

@Jan21, any guesses?
Low priority, but if you see a quick/easy fix, let me know and I'll give it a try.
You had mnli in train_tasks? I don't know why the eval loop works this way, but it evaluates train tasks too...if you have glue for train and for eval, then it will evaluate them 2 times...I guess that if you had mnli in train_tasks then it will be added to eval_tasks because of the "use_classifier" logic...I don't know how it works exactly. But if you add mnli to eval_tasks then it will be evaluated only once I guess.
Thanks! That helps.

On Mon, Jul 23, 2018 at 6:50 PM Jan Hůla <notifications@github.com> wrote:

> You had mnli in train_tasks? I don't know why the eval loop works this
> way, but it evaluates train tasks too...if you have glue for train and for
> eval, then it will evaluate them 2 times...I guess that if you had mnli in
> train_tasks then it will be added to eval_tasks because of the
> "use_classifier" logic...I don't know how it works exactly. But if you add
> mnli to eval_tasks then it will be evaluated only once I guess.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/jsalt18-sentence-repl/jiant/issues/232#issuecomment-407225615>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABOZWfbjHHdezRO3Dxxn18NeYpVw-ubJks5uJlNEgaJpZM4VbzPo>
> .
>

Is it possible that there are two different MNLI tasks with the same name?

Resolving patch:
diff --git a/main.py b/main.py
--- a/main.py
+++ b/main.py
@@ -269,16 +269,16 @@ def main(cl_arguments):
     if args.do_eval:
         # Evaluate #
         log.info("Evaluating...")
-        val_results, val_preds = evaluate.evaluate(model, tasks,
+        val_results, val_preds = evaluate.evaluate(model, eval_tasks,
                                                    args.batch_size,
                                                    args.cuda, "val")
 
         splits_to_write = evaluate.parse_write_preds_arg(args.write_preds)
         if 'val' in splits_to_write:
-            evaluate.write_preds(tasks, val_preds, args.run_dir, 'val',
+            evaluate.write_preds(eval_tasks, val_preds, args.run_dir, 'val',
                                  strict_glue_format=args.write_strict_glue_format)
         if 'test' in splits_to_write:
-            _, te_preds = evaluate.evaluate(model, tasks,
+            _, te_preds = evaluate.evaluate(model, eval_tasks,
                                             args.batch_size, args.cuda, "test")
             evaluate.write_preds(tasks, te_preds, args.run_dir, 'test',
                                  strict_glue_format=args.write_strict_glue_format)


Test patch: