Repository: mlcommons/training (language: python)

Issue:
Transformer
Might need a few small tweaks depending on how current runs go. But this is more or less the final form. 


Discussion / hints:

Fix patch:
diff --git a/.gitmodules b/.gitmodules
index 3eebde865..7df4596d3 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,9 +1,3 @@
-[submodule "community"]
-	path = community
-	url = https://github.com/mlperf/community.git
-[submodule "compliance/verify_submission/mlperf_submission_helper/mlp_compliance"]
-	path = compliance/verify_submission/mlperf_submission_helper/mlp_compliance
-	url = https://github.com/bitfort/mlp_compliance
 [submodule "recommendation/dlrm"]
 	path = recommendation/dlrm
 	url = https://github.com/facebookresearch/dlrm.git
diff --git a/community b/community
deleted file mode 160000
index eebf719c1..000000000
--- a/community
+++ /dev/null
@@ -1,1 +0,0 @@
-Subproject commit eebf719c1b2174243c5d275dc57acc045bb3722d
diff --git a/compliance/LICENSE.md b/compliance/LICENSE.md
deleted file mode 100644
index 89ba5be48..000000000
--- a/compliance/LICENSE.md
+++ /dev/null
@@ -1,201 +0,0 @@
-                                 Apache License
-                           Version 2.0, January 2004
-                        http://www.apache.org/licenses/
-
-   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
-
-   1. Definitions.
-
-      "License" shall mean the terms and conditions for use, reproduction,
-      and distribution as defined by Sections 1 through 9 of this document.
-
-      "Licensor" shall mean the copyright owner or entity authorized by
-      the copyright owner that is granting the License.
-
-      "Legal Entity" shall mean the union of the acting entity and all
-      other entities that control, are controlled by, or are under common
-      control with that entity. For the purposes of this definition,
-      "control" means (i) the power, direct or indirect, to cause the
-      direction or management of such entity, whether by contract or
-      otherwise, or (ii) ownership of fifty percent (50%) or more of the
-      outstanding shares, or (iii) beneficial ownership of such entity.
-
-      "You" (or "Your") shall mean an individual or Legal Entity
-      exercising permissions granted by this License.
-
-      "Source" form shall mean the preferred form for making modifications,
-      including but not limited to software source code, documentation
-      source, and configuration files.
-
-      "Object" form shall mean any form resulting from mechanical
-      transformation or translation of a Source form, including but
-      not limited to compiled object code, generated documentation,
-      and conversions to other media types.
-
-      "Work" shall mean the work of authorship, whether in Source or
-      Object form, made available under the License, as indicated by a
-      copyright notice that is included in or attached to the work
-      (an example is provided in the Appendix below).
-
-      "Derivative Works" shall mean any work, whether in Source or Object
-      form, that is based on (or derived from) the Work and for which the
-      editorial revisions, annotations, elaborations, or other modifications
-      represent, as a whole, an original work of authorship. For the purposes
-      of this License, Derivative Works shall not include works that remain
-      separable from, or merely link (or bind by name) to the interfaces of,
-      the Work and Derivative Works thereof.
-
-      "Contribution" shall mean any work of authorship, including
-      the original version of the Work and any modifications or additions
-      to that Work or Derivative Works thereof, that is intentionally
-      submitted to Licensor for inclusion in the Work by the copyright owner
-      or by an individual or Legal Entity authorized to submit on behalf of
-      the copyright owner. For the purposes of this definition, "submitted"
-      means any form of electronic, verbal, or written communication sent
-      to the Licensor or its representatives, including but not limited to
-      communication on electronic mailing lists, source code control systems,
-      and issue tracking systems that are managed by, or on behalf of, the
-      Licensor for the purpose of discussing and improving the Work, but
-      excluding communication that is conspicuously marked or otherwise
-      designated in writing by the copyright owner as "Not a Contribution."
-
-      "Contributor" shall mean Licensor and any individual or Legal Entity
-      on behalf of whom a Contribution has been received by Licensor and
-      subsequently incorporated within the Work.
-
-   2. Grant of Copyright License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      copyright license to reproduce, prepare Derivative Works of,
-      publicly display, publicly perform, sublicense, and distribute the
-      Work and such Derivative Works in Source or Object form.
-
-   3. Grant of Patent License. Subject to the terms and conditions of
-      this License, each Contributor hereby grants to You a perpetual,
-      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
-      (except as stated in this section) patent license to make, have made,
-      use, offer to sell, sell, import, and otherwise transfer the Work,
-      where such license applies only to those patent claims licensable
-      by such Contributor that are necessarily infringed by their
-      Contribution(s) alone or by combination of their Contribution(s)
-      with the Work to which such Contribution(s) was submitted. If You
-      institute patent litigation against any entity (including a
-      cross-claim or counterclaim in a lawsuit) alleging that the Work
-      or a Contribution incorporated within the Work constitutes direct
-      or contributory patent infringement, then any patent licenses
-      granted to You under this License for that Work shall terminate
-      as of the date such litigation is filed.
-
-   4. Redistribution. You may reproduce and distribute copies of the
-      Work or Derivative Works thereof in any medium, with or without
-      modifications, and in Source or Object form, provided that You
-      meet the following conditions:
-
-      (a) You must give any other recipients of the Work or
-          Derivative Works a copy of this License; and
-
-      (b) You must cause any modified files to carry prominent notices
-          stating that You changed the files; and
-
-      (c) You must retain, in the Source form of any Derivative Works
-          that You distribute, all copyright, patent, trademark, and
-          attribution notices from the Source form of the Work,
-          excluding those notices that do not pertain to any part of
-          the Derivative Works; and
-
-      (d) If the Work includes a "NOTICE" text file as part of its
-          distribution, then any Derivative Works that You distribute must
-          include a readable copy of the attribution notices contained
-          within such NOTICE file, excluding those notices that do not
-          pertain to any part of the Derivative Works, in at least one
-          of the following places: within a NOTICE text file distributed
-          as part of the Derivative Works; within the Source form or
-          documentation, if provided along with the Derivative Works; or,
-          within a display generated by the Derivative Works, if and
-          wherever such third-party notices normally appear. The contents
-          of the NOTICE file are for informational purposes only and
-          do not modify the License. You may add Your own attribution
-          notices within Derivative Works that You distribute, alongside
-          or as an addendum to the NOTICE text from the Work, provided
-          that such additional attribution notices cannot be construed
-          as modifying the License.
-
-      You may add Your own copyright statement to Your modifications and
-      may provide additional or different license terms and conditions
-      for use, reproduction, or distribution of Your modifications, or
-      for any such Derivative Works as a whole, provided Your use,
-      reproduction, and distribution of the Work otherwise complies with
-      the conditions stated in this License.
-
-   5. Submission of Contributions. Unless You explicitly state otherwise,
-      any Contribution intentionally submitted for inclusion in the Work
-      by You to the Licensor shall be under the terms and conditions of
-      this License, without any additional terms or conditions.
-      Notwithstanding the above, nothing herein shall supersede or modify
-      the terms of any separate license agreement you may have executed
-      with Licensor regarding such Contributions.
-
-   6. Trademarks. This License does not grant permission to use the trade
-      names, trademarks, service marks, or product names of the Licensor,
-      except as required for reasonable and customary use in describing the
-      origin of the Work and reproducing the content of the NOTICE file.
-
-   7. Disclaimer of Warranty. Unless required by applicable law or
-      agreed to in writing, Licensor provides the Work (and each
-      Contributor provides its Contributions) on an "AS IS" BASIS,
-      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
-      implied, including, without limitation, any warranties or conditions
-      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
-      PARTICULAR PURPOSE. You are solely responsible for determining the
-      appropriateness of using or redistributing the Work and assume any
-      risks associated with Your exercise of permissions under this License.
-
-   8. Limitation of Liability. In no event and under no legal theory,
-      whether in tort (including negligence), contract, or otherwise,
-      unless required by applicable law (such as deliberate and grossly
-      negligent acts) or agreed to in writing, shall any Contributor be
-      liable to You for damages, including any direct, indirect, special,
-      incidental, or consequential damages of any character arising as a
-      result of this License or out of the use or inability to use the
-      Work (including but not limited to damages for loss of goodwill,
-      work stoppage, computer failure or malfunction, or any and all
-      other commercial damages or losses), even if such Contributor
-      has been advised of the possibility of such damages.
-
-   9. Accepting Warranty or Additional Liability. While redistributing
-      the Work or Derivative Works thereof, You may choose to offer,
-      and charge a fee for, acceptance of support, warranty, indemnity,
-      or other liability obligations and/or rights consistent with this
-      License. However, in accepting such obligations, You may act only
-      on Your own behalf and on Your sole responsibility, not on behalf
-      of any other Contributor, and only if You agree to indemnify,
-      defend, and hold each Contributor harmless for any liability
-      incurred by, or claims asserted against, such Contributor by reason
-      of your accepting any such warranty or additional liability.
-
-   END OF TERMS AND CONDITIONS
-
-   APPENDIX: How to apply the Apache License to your work.
-
-      To apply the Apache License to your work, attach the following
-      boilerplate notice, with the fields enclosed by brackets "[]"
-      replaced with your own identifying information. (Don't include
-      the brackets!)  The text should be enclosed in the appropriate
-      comment syntax for the file format. We also recommend that a
-      file or class name and description of purpose be included on the
-      same "printed page" as the copyright notice for easier
-      identification within third-party archives.
-
-   Copyright 2018 The MLPerf Authors
-
-   Licensed under the Apache License, Version 2.0 (the "License");
-   you may not use this file except in compliance with the License.
-   You may obtain a copy of the License at
-
-       http://www.apache.org/licenses/LICENSE-2.0
-
-   Unless required by applicable law or agreed to in writing, software
-   distributed under the License is distributed on an "AS IS" BASIS,
-   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-   See the License for the specific language governing permissions and
-   limitations under the License.
diff --git a/compliance/README.md b/compliance/README.md
deleted file mode 100644
index 5c952f121..000000000
--- a/compliance/README.md
+++ /dev/null
@@ -1,1 +0,0 @@
-# MLPerf Compliance Logging Utilities and Helper Functions
\ No newline at end of file
diff --git a/compliance/example_logs/gnmt_example.log b/compliance/example_logs/gnmt_example.log
deleted file mode 100644
index 89a6233c0..000000000
--- a/compliance/example_logs/gnmt_example.log
+++ /dev/null
@@ -1,74 +0,0 @@
-:::MLPv0.5.0 gnmt 1550049265.393176317 (train.py:268) run_start
-:::MLPv0.5.0 gnmt 1550049265.394439697 (seq2seq/utils.py:113) run_set_random_seed: 1
-:::MLPv0.5.0 gnmt 1550049265.412598848 (train.py:312) preproc_tokenize_training
-:::MLPv0.5.0 gnmt 1550049265.413046360 (train.py:314) train_hp_max_sequence_length: 50
-:::MLPv0.5.0 gnmt 1550049274.456599236 (train.py:326) preproc_num_train_examples: 3498161
-:::MLPv0.5.0 gnmt 1550049275.559447765 (train.py:336) preproc_tokenize_eval
-:::MLPv0.5.0 gnmt 1550049275.597178459 (train.py:346) preproc_num_eval_examples: 3003
-:::MLPv0.5.0 gnmt 1550049275.597592831 (train.py:350) preproc_vocab_size: 32317
-:::MLPv0.5.0 gnmt 1550049275.598169804 (seq2seq/models/gnmt.py:34) model_hp_num_layers: 4
-:::MLPv0.5.0 gnmt 1550049275.598643303 (seq2seq/models/gnmt.py:36) model_hp_hidden_size: 1024
-:::MLPv0.5.0 gnmt 1550049275.599111080 (seq2seq/models/gnmt.py:38) model_hp_dropout: 0.2
-:::MLPv0.5.0 gnmt 1550049279.603173733 (train.py:251) model_hp_loss_fn: "Cross Entropy with label smoothing"
-:::MLPv0.5.0 gnmt 1550049279.603722811 (train.py:253) model_hp_loss_smoothing: 0.1
-:::MLPv0.5.0 gnmt 1550049280.493244410 (train.py:393) input_batch_size: 128
-:::MLPv0.5.0 gnmt 1550049280.493689060 (train.py:395) input_size: 3497728
-:::MLPv0.5.0 gnmt 1550049280.494647741 (seq2seq/data/sampler.py:254) input_order
-:::MLPv0.5.0 gnmt 1550049280.496007442 (seq2seq/data/sampler.py:254) input_order
-:::MLPv0.5.0 gnmt 1550049280.496948242 (train.py:409) eval_size: 3003
-:::MLPv0.5.0 gnmt 1550049280.497889280 (seq2seq/inference/beam_search.py:43) eval_hp_beam_size: 5
-:::MLPv0.5.0 gnmt 1550049280.498333216 (seq2seq/inference/beam_search.py:45) eval_hp_max_sequence_length: 150
-:::MLPv0.5.0 gnmt 1550049280.498714685 (seq2seq/inference/beam_search.py:47) eval_hp_length_normalization_constant: 5.0
-:::MLPv0.5.0 gnmt 1550049280.499088049 (seq2seq/inference/beam_search.py:49) eval_hp_length_normalization_factor: 0.6
-:::MLPv0.5.0 gnmt 1550049280.499464273 (seq2seq/inference/beam_search.py:51) eval_hp_coverage_penalty_factor: 0.1
-:::MLPv0.5.0 gnmt 1550049283.242700577 (seq2seq/train/trainer.py:115) opt_name: "adam"
-:::MLPv0.5.0 gnmt 1550049283.243218422 (seq2seq/train/trainer.py:117) opt_learning_rate: 0.001
-:::MLPv0.5.0 gnmt 1550049283.243656635 (seq2seq/train/trainer.py:119) opt_hp_Adam_beta1: 0.9
-:::MLPv0.5.0 gnmt 1550049283.244005442 (seq2seq/train/trainer.py:121) opt_hp_Adam_beta2: 0.999
-:::MLPv0.5.0 gnmt 1550049283.244350195 (seq2seq/train/trainer.py:123) opt_hp_Adam_epsilon: 1e-08
-:::MLPv0.5.0 gnmt 1550049283.245311022 (seq2seq/train/lr_scheduler.py:78) opt_learning_rate_warmup_steps: 200
-:::MLPv0.5.0 gnmt 1550049283.245801687 (train.py:466) train_loop
-:::MLPv0.5.0 gnmt 1550049283.246350050 (train.py:470) train_epoch: 0
-:::MLPv0.5.0 gnmt 1550049284.053482771 (seq2seq/data/sampler.py:205) input_order
-:::MLPv0.5.0 gnmt 1550061484.477649927 (train.py:483) train_checkpoint
-:::MLPv0.5.0 gnmt 1550061486.181115627 (train.py:490) eval_start: 0
-:::MLPv0.5.0 gnmt 1550061514.467880964 (train.py:495) eval_accuracy: {"epoch": 0, "value": 20.49}
-:::MLPv0.5.0 gnmt 1550061514.468363047 (train.py:497) eval_target: 24.0
-:::MLPv0.5.0 gnmt 1550061514.468725204 (train.py:498) eval_stop
-:::MLPv0.5.0 gnmt 1550061514.469318390 (train.py:470) train_epoch: 1
-:::MLPv0.5.0 gnmt 1550061515.311945915 (seq2seq/data/sampler.py:205) input_order
-:::MLPv0.5.0 gnmt 1550073699.708187580 (train.py:483) train_checkpoint
-:::MLPv0.5.0 gnmt 1550073705.950762510 (train.py:490) eval_start: 1
-:::MLPv0.5.0 gnmt 1550073733.111917734 (train.py:495) eval_accuracy: {"epoch": 1, "value": 21.72}
-:::MLPv0.5.0 gnmt 1550073733.112442732 (train.py:497) eval_target: 24.0
-:::MLPv0.5.0 gnmt 1550073733.112777948 (train.py:498) eval_stop
-:::MLPv0.5.0 gnmt 1550073733.113427401 (train.py:470) train_epoch: 2
-:::MLPv0.5.0 gnmt 1550073733.976921797 (seq2seq/data/sampler.py:205) input_order
-:::MLPv0.5.0 gnmt 1550085915.256659985 (train.py:483) train_checkpoint
-:::MLPv0.5.0 gnmt 1550085921.003725052 (train.py:490) eval_start: 2
-:::MLPv0.5.0 gnmt 1550085948.763688564 (train.py:495) eval_accuracy: {"epoch": 2, "value": 22.52}
-:::MLPv0.5.0 gnmt 1550085948.764161110 (train.py:497) eval_target: 24.0
-:::MLPv0.5.0 gnmt 1550085948.764480591 (train.py:498) eval_stop
-:::MLPv0.5.0 gnmt 1550085948.765091658 (train.py:470) train_epoch: 3
-:::MLPv0.5.0 gnmt 1550085949.625080585 (seq2seq/data/sampler.py:205) input_order
-:::MLPv0.5.0 gnmt 1550098119.481947660 (train.py:483) train_checkpoint
-:::MLPv0.5.0 gnmt 1550098125.219326258 (train.py:490) eval_start: 3
-:::MLPv0.5.0 gnmt 1550098154.012802124 (train.py:495) eval_accuracy: {"epoch": 3, "value": 23.09}
-:::MLPv0.5.0 gnmt 1550098154.013273716 (train.py:497) eval_target: 24.0
-:::MLPv0.5.0 gnmt 1550098154.013695478 (train.py:498) eval_stop
-:::MLPv0.5.0 gnmt 1550098154.014324665 (train.py:470) train_epoch: 4
-:::MLPv0.5.0 gnmt 1550098154.874188900 (seq2seq/data/sampler.py:205) input_order
-:::MLPv0.5.0 gnmt 1550110332.296045065 (train.py:483) train_checkpoint
-:::MLPv0.5.0 gnmt 1550110338.542354584 (train.py:490) eval_start: 4
-:::MLPv0.5.0 gnmt 1550110366.501193285 (train.py:495) eval_accuracy: {"epoch": 4, "value": 23.22}
-:::MLPv0.5.0 gnmt 1550110366.501719475 (train.py:497) eval_target: 24.0
-:::MLPv0.5.0 gnmt 1550110366.502229214 (train.py:498) eval_stop
-:::MLPv0.5.0 gnmt 1550110366.502951622 (train.py:470) train_epoch: 5
-:::MLPv0.5.0 gnmt 1550110367.328888416 (seq2seq/data/sampler.py:205) input_order
-:::MLPv0.5.0 gnmt 1550122538.255764246 (train.py:483) train_checkpoint
-:::MLPv0.5.0 gnmt 1550122544.367511034 (train.py:490) eval_start: 5
-:::MLPv0.5.0 gnmt 1550122571.801926613 (train.py:495) eval_accuracy: {"epoch": 5, "value": 24.11}
-:::MLPv0.5.0 gnmt 1550122571.802411556 (train.py:497) eval_target: 24.0
-:::MLPv0.5.0 gnmt 1550122571.802731991 (train.py:498) eval_stop
-:::MLPv0.5.0 gnmt 1550122571.803371668 (train.py:522) run_stop: {"success": true}
-:::MLPv0.5.0 gnmt 1550122571.803844213 (train.py:523) run_final
diff --git a/compliance/example_logs/ncf_example.log b/compliance/example_logs/ncf_example.log
deleted file mode 100644
index 0fde05361..000000000
--- a/compliance/example_logs/ncf_example.log
+++ /dev/null
@@ -1,252 +0,0 @@
-Start processing ml-20m/ratings.csv
-Loading raw data from ml-20m/ratings.csv
-20000263 ratings on 26744 items from 138493 users from 1995-01-09 11:46:44 to 2015-03-31 06:40:02
-Filtering out users with less than 20 ratings
-
-:::MLPv0.5.0 ncf 1538677896 (./convert.py:119) preproc_hp_min_ratings: 20
-Mapping original user and item IDs to new sequential IDs
-Creating list of items for each user
-Ratings: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 20000263/20000263 [00:26<00:00, 767559.37it/s]
-Generating 999 negative samples for each user
-
-:::MLPv0.5.0 ncf 1538678136 (./convert.py:119) preproc_hp_num_eval: 999
-
-:::MLPv0.5.0 ncf 1538678136 (./convert.py:119) preproc_hp_sample_eval_replacement: true
-
-:::MLPv0.5.0 ncf 1538678136 (./convert.py:119) run_start
-
-:::MLPv0.5.0 ncf 1538678136 (./convert.py:119) input_step_eval_neg_gen
-Users: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 138493/138493 [06:13<00:00, 371.04it/s]
-Saving train and test CSV files to ml-20m
-
-:::MLPv0.5.0 ncf 1538678620 (./convert.py:119) input_size: 19861770
-Finish processing ml-20m/ratings.csv in 1359 seconds
-Start training
-Using seed = 1
-Saving config and results to ./run/neumf/1538679245
-Loading data
-
-:::MLPv0.5.0 ncf 1538679509 (./ncf.py:173) input_step_train_neg_gen: 4
-
-:::MLPv0.5.0 ncf 1538679509 (./ncf.py:173) input_hp_sample_train_replacement
-
-:::MLPv0.5.0 ncf 1538679509 (./ncf.py:284) input_batch_size: 2048
-
-:::MLPv0.5.0 ncf 1538679509 (./ncf.py:284) input_order
-Load data done [298.2 s]. #user=138493, #item=26744, #train=19861770, #test=138493
-
-:::MLPv0.5.0 ncf 1538679542 (./ncf.py:191) model_hp_mf_dim
-
-:::MLPv0.5.0 ncf 1538679543 (./ncf.py:191) model_hp_mlp_layer_sizes: [256, 256, 128, 64]
-NeuMF(
-  (mf_user_embed): Embedding(138493, 64)
-  (mf_item_embed): Embedding(26744, 64)
-  (mlp_user_embed): Embedding(138493, 128)
-  (mlp_item_embed): Embedding(26744, 128)
-  (mlp): ModuleList(
-    (0): Linear(in_features=256, out_features=256, bias=True)
-    (1): Linear(in_features=256, out_features=128, bias=True)
-    (2): Linear(in_features=128, out_features=64, bias=True)
-  )
-  (final): Linear(in_features=128, out_features=1, bias=True)
-)
-31832577 parameters
-
-:::MLPv0.5.0 ncf 1538679544 (./ncf.py:284) train_learn_rate: 0.0005
-
-:::MLPv0.5.0 ncf 1538679544 (./ncf.py:284) opt_name: "Adam"
-
-:::MLPv0.5.0 ncf 1538679544 (./ncf.py:284) opt_hp_Adam_beta1: 0.9
-
-:::MLPv0.5.0 ncf 1538679544 (./ncf.py:284) opt_hp_Adam_beta2: 0.999
-
-:::MLPv0.5.0 ncf 1538679544 (./ncf.py:284) opt_hp_Adam_epsilon: 1e-08
-
-:::MLPv0.5.0 ncf 1538679544 (./ncf.py:284) model_hp_loss_fn: "binary_cross_entropy"
-Initial evaluation
-
-:::MLPv0.5.0 ncf 1538679548 (./ncf.py:222) eval_start
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:222) eval_size: {"epoch": null, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:222) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:222) eval_hp_num_neg: 999
-Initial HR@10 = 0.0105, NDCG@10 = 0.0054
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:284) train_loop
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:284) train_epoch: 0
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538679634 (./ncf.py:284) input_step_train_neg_gen
-Epoch 0 Loss 0.1237 (0.1468): 100%|█████████████████████████████████████████████████████████████████████████████████████| 48491/48491 [10:24<00:00, 77.69it/s]
-Epoch 0 evaluation
-
-:::MLPv0.5.0 ncf 1538680258 (./ncf.py:263) eval_start: 0
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:263) eval_size: {"epoch": 0, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:284) eval_accuracy: {"epoch": 0, "value": 0.5749676823616028}
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:284) eval_stop
-Epoch 0: HR@10 = 0.5750, NDCG@10 = 0.3379, train_time = 624.15, val_time = 86.71
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:284) train_epoch: 1
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538680345 (./ncf.py:284) input_step_train_neg_gen
-Epoch 1 Loss 0.1162 (0.1170): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
-██████████████| 48491/48491 [06:30<00:00, 124.08it/s]
-Epoch 1 evaluation
-
-:::MLPv0.5.0 ncf 1538680736 (./ncf.py:263) eval_start: 1
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:263) eval_size: {"epoch": 1, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:284) eval_accuracy: {"epoch": 1, "value": 0.6050702929496765}
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:284) eval_stop
-Epoch 1: HR@10 = 0.6051, NDCG@10 = 0.3565, train_time = 390.80, val_time = 87.01
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:284) train_epoch: 2
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538680823 (./ncf.py:284) input_step_train_neg_gen
-Epoch 2 Loss 0.1082 (0.1083): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
-██████████████| 48491/48491 [06:30<00:00, 124.08it/s]
-Epoch 2 evaluation
-
-:::MLPv0.5.0 ncf 1538681214 (./ncf.py:263) eval_start: 2
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:263) eval_size: {"epoch": 2, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:284) eval_accuracy: {"epoch": 2, "value": 0.6190782189369202}
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:284) eval_stop
-Epoch 2: HR@10 = 0.6191, NDCG@10 = 0.3667, train_time = 390.81, val_time = 86.57
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:284) train_epoch: 3
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538681300 (./ncf.py:284) input_step_train_neg_gen
-Epoch 3 Loss 0.1041 (0.1029): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
-██████████████| 48491/48491 [06:30<00:00, 124.21it/s]
-Epoch 3 evaluation
-
-:::MLPv0.5.0 ncf 1538681691 (./ncf.py:263) eval_start: 3
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:263) eval_size: {"epoch": 3, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:284) eval_accuracy: {"epoch": 3, "value": 0.6252879500389099}
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:284) eval_stop
-Epoch 3: HR@10 = 0.6253, NDCG@10 = 0.3724, train_time = 390.40, val_time = 86.90
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:284) train_epoch: 4
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538681778 (./ncf.py:284) input_step_train_neg_gen
-Epoch 4 Loss 0.0932 (0.0993): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
-██████████████| 48491/48491 [06:36<00:00, 122.23it/s]
-Epoch 4 evaluation
-
-:::MLPv0.5.0 ncf 1538682174 (./ncf.py:263) eval_start: 4
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:263) eval_size: {"epoch": 4, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:284) eval_accuracy: {"epoch": 4, "value": 0.6261760592460632}
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:284) eval_stop
-Epoch 4: HR@10 = 0.6262, NDCG@10 = 0.3736, train_time = 396.72, val_time = 87.04
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:284) train_epoch: 5
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538682261 (./ncf.py:284) input_step_train_neg_gen
-Epoch 5 Loss 0.0999 (0.0965): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
-██████████████| 48491/48491 [06:30<00:00, 124.15it/s]
-Epoch 5 evaluation
-
-:::MLPv0.5.0 ncf 1538682652 (./ncf.py:263) eval_start: 5
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:263) eval_size: {"epoch": 5, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:284) eval_accuracy: {"epoch": 5, "value": 0.6341837048530579}
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:284) eval_stop
-Epoch 5: HR@10 = 0.6342, NDCG@10 = 0.3805, train_time = 390.60, val_time = 87.13
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:284) train_epoch: 6
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:284) input_hp_num_neg: 4
-
-:::MLPv0.5.0 ncf 1538682739 (./ncf.py:284) input_step_train_neg_gen
-Epoch 6 Loss 0.0888 (0.0944): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
-██████████████| 48491/48491 [06:30<00:00, 124.02it/s]
-Epoch 6 evaluation
-
-:::MLPv0.5.0 ncf 1538683130 (./ncf.py:263) eval_start: 6
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:263) eval_size: {"epoch": 6, "value": 138493000}
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:263) eval_hp_num_users: 138493
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:263) eval_hp_num_neg: 999
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:284) eval_accuracy: {"epoch": 6, "value": 0.6359454989433289}
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:284) eval_target: 0.635
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:284) eval_stop
-Epoch 6: HR@10 = 0.6359, NDCG@10 = 0.3825, train_time = 390.98, val_time = 86.88
-Hit threshold of 0.635
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:284) run_stop: {"success": true}
-
-:::MLPv0.5.0 ncf 1538683217 (./ncf.py:284) run_final
-Finish training in 3987 seconds
-ENDING TIMING RUN AT 2018-10-04 08:00:31 PM
-RESULT,recommendation,1,5347,taylorrobie,2018-10-04 06:31:24 PM
diff --git a/compliance/example_logs/resnet_example.log b/compliance/example_logs/resnet_example.log
deleted file mode 100644
index c4cf2cee6..000000000
--- a/compliance/example_logs/resnet_example.log
+++ /dev/null
@@ -1,1202 +0,0 @@
-:::MLPv0.5.0 resnet 1539716218.578738451 (official/resnet/imagenet_main.py:328) run_set_random_seed: 1539716216
-:::MLPv0.5.0 resnet 1539716218.593775988 (official/resnet/imagenet_main.py:334) preproc_num_train_examples: 1281167
-:::MLPv0.5.0 resnet 1539716218.594253540 (official/resnet/imagenet_main.py:336) preproc_num_eval_examples: 50000
-:::MLPv0.5.0 resnet 1539716218.596412420 (resnet_run_loop.py:386) run_start
-:::MLPv0.5.0 resnet 1539716221.922842503 (resnet_run_loop.py:409) run_set_random_seed: 1539716216
-:::MLPv0.5.0 resnet 1539716221.923823118 (resnet_run_loop.py:415) input_batch_size: 64
-:::MLPv0.5.0 resnet 1539716221.925445318 (resnet_run_loop.py:433) train_loop
-:::MLPv0.5.0 resnet 1539716221.925973892 (resnet_run_loop.py:436) eval_offset: 3
-:::MLPv0.5.0 resnet 1539716221.926461697 (resnet_run_loop.py:445) train_epoch: 0
-:::MLPv0.5.0 resnet 1539716221.926892757 (resnet_run_loop.py:445) train_epoch: 1
-:::MLPv0.5.0 resnet 1539716221.927357674 (resnet_run_loop.py:445) train_epoch: 2
-:::MLPv0.5.0 resnet 1539716221.927788973 (resnet_run_loop.py:445) train_epoch: 3
-:::MLPv0.5.0 resnet 1539716221.930120707 (official/resnet/imagenet_main.py:177) input_order
-:::MLPv0.5.0 resnet 1539716221.964905262 (resnet_run_loop.py:76) input_order
-:::MLPv0.5.0 resnet 1539716221.994927168 (imagenet_preprocessing.py:86) input_distorted_crop_min_object_covered: 0.1
-:::MLPv0.5.0 resnet 1539716221.996390104 (imagenet_preprocessing.py:88) input_distorted_crop_aspect_ratio_range: [0.75, 1.33]
-:::MLPv0.5.0 resnet 1539716221.997827530 (imagenet_preprocessing.py:90) input_distorted_crop_area_range: [0.05, 1.0]
-:::MLPv0.5.0 resnet 1539716221.999229193 (imagenet_preprocessing.py:92) input_distorted_crop_max_attempts: 100
-:::MLPv0.5.0 resnet 1539716222.008083820 (imagenet_preprocessing.py:114) input_random_flip
-:::MLPv0.5.0 resnet 1539716222.030642271 (imagenet_preprocessing.py:281) input_resize: [224, 224]
-:::MLPv0.5.0 resnet 1539716222.036507368 (imagenet_preprocessing.py:173) input_mean_subtraction: [123.68, 116.78, 103.94]
-:::MLPv0.5.0 resnet 1539716222.074727058 (resnet_model.py:396) model_hp_initial_shape: [224, 224, 3]
-:::MLPv0.5.0 resnet 1539716222.093446732 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(3, 224, 224) -> (64, 112, 112)"
-:::MLPv0.5.0 resnet 1539716222.094428062 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716222.138671398 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 112, 112)"}
-:::MLPv0.5.0 resnet 1539716222.139570236 (resnet_model.py:417) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.143005371 (resnet_model.py:425) model_hp_initial_max_pool: "(64, 112, 112) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.144744873 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716222.145790100 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.161190271 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.162257671 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.163206100 (resnet_model.py:164) model_hp_projection_shortcut: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.205909252 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.220429659 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.221447945 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.264477491 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.265417576 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.280450106 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.281468630 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.384546757 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.385513544 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.400581360 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.401613235 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.443728447 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.444657564 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716222.446455717 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.448173523 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.449147463 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716222.450125217 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.464468241 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.465490103 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.508072376 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.508998394 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.524943829 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.525987864 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.568631649 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.569556713 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.584595919 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.585607290 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.628346920 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.629304886 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716222.631071329 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.632890940 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.633893251 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716222.634854317 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.649195194 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.650241613 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.693548679 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.694498777 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.709828854 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.710850477 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.753895044 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.754819632 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.769931555 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.770949125 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.814394474 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.815311909 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716222.817076921 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.818874598 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.820597649 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716222.821622849 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.837870836 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716222.838929415 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716222.839896441 (resnet_model.py:164) model_hp_projection_shortcut: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716222.882608175 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716222.897049189 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (128, 56, 56)"
-:::MLPv0.5.0 resnet 1539716222.898092270 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716222.942653179 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716222.943666697 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716222.962580681 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 56, 56) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716222.963628531 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716223.010366440 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.011314154 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.026599884 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.027622700 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.070532322 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.071478367 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716223.073220253 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.075000763 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.075982094 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716223.076934814 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.091453314 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.092479706 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.135893106 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.136835575 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.151966095 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.152988195 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.196076632 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.197013855 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.212255478 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.213275433 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.256017208 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.256945133 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716223.258733749 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.260495186 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.261471510 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716223.262458563 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.277568579 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.278611898 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.321444273 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.322396040 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.337502956 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.338547707 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.381501675 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.382474661 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.397617579 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.398683786 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.442117691 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.443057537 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716223.444823742 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.446601868 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.447579384 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716223.448534489 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.463047266 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.464065552 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.506669283 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.507604122 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.523007631 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.524028540 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.567697763 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.568667650 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.584035397 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.585055828 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.627993584 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.628925562 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716223.630698442 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.632466555 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.634173870 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716223.635189295 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.651675701 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.652785778 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716223.653764963 (resnet_model.py:164) model_hp_projection_shortcut: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.703101635 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716223.718224525 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (256, 28, 28)"
-:::MLPv0.5.0 resnet 1539716223.719272375 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.762155294 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716223.763093472 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.780062437 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 28, 28) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.781115770 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716223.824270725 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716223.825192451 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.840514183 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.841591120 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.891871214 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716223.892798185 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716223.894568443 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.896324396 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.897274733 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716223.898253918 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.912710190 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.913754940 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716223.957176685 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716223.958126307 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716223.973499060 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716223.974553823 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.093811512 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.094774008 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.110135555 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.111147404 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.159577608 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.160475731 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716224.162206411 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.163934708 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.164934635 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716224.165909290 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.180010557 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.181014061 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.223010063 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.223923922 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.239379168 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.240430355 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.282536745 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.283447742 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.298369646 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.299370050 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.347666025 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.348567963 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716224.350314379 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.352008820 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.352964878 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716224.353932142 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.368031502 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.369036674 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.411758184 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.412714481 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.427826405 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.428832293 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.470839500 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.471763611 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.486670732 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.487672329 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.536700010 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.537607908 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716224.539354086 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.541048765 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.542036533 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716224.542975664 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.557097435 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.558129072 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.600381374 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.601269484 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.616153002 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.617155790 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.659827709 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.660742521 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.675638199 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.676641941 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.725033283 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.726037979 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716224.727770805 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.729488611 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.730472565 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716224.731421232 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.745685339 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.746714830 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.789170265 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.790143967 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.805758238 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.806803942 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.849216461 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.850150347 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.865120649 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.866153717 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716224.914574385 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716224.915482759 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716224.917444468 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716224.919173956 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.920877695 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716224.921915531 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716224.938043356 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716224.939099789 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716224.940046787 (resnet_model.py:164) model_hp_projection_shortcut: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716224.989267588 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.003496885 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (512, 14, 14)"
-:::MLPv0.5.0 resnet 1539716225.004495859 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.046848774 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716225.047886133 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.064568758 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 14, 14) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.065577507 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716225.108498573 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.109404325 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.124269247 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.125269651 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.173898935 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.174812555 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716225.176550865 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.178304195 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.179248333 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716225.180185556 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.194452047 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.195465565 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.238263130 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.239169598 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.254103661 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.255110264 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.297062635 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.298014164 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.312940598 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.313969612 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.363301039 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.364196062 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716225.365940332 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.367683887 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.368643284 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716225.369593143 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.383817673 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.384813070 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.427322149 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.428239107 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.443462133 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.444468021 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.486741304 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.487655878 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.502685547 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.503691673 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716225.552781820 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716225.553723812 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716225.555464506 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716225.557169914 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716225.562492609 (resnet_model.py:455) model_hp_dense: 1001
-:::MLPv0.5.0 resnet 1539716225.585315704 (resnet_model.py:461) model_hp_final_shape: [1001]
-:::MLPv0.5.0 resnet 1539716225.601087809 (resnet_run_loop.py:249) model_hp_loss_fn: "categorical_cross_entropy"
-:::MLPv0.5.0 resnet 1539716225.636411190 (resnet_run_loop.py:264) model_exclude_bn_from_l2: true
-:::MLPv0.5.0 resnet 1539716225.637198925 (resnet_run_loop.py:268) model_l2_regularization: 0.0001
-:::MLPv0.5.0 resnet 1539716225.940773487 (resnet_run_loop.py:290) opt_name: "stochastic_gradient_descent_with_momentum"
-:::MLPv0.5.0 resnet 1539716225.941622257 (resnet_run_loop.py:291) opt_momentum: 0.9
-:::MLPv0.5.0 resnet 1539716240.4685431 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 0.0}
-:::MLPv0.5.0 resnet 1539716276.769443 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 3.1970907e-05}
-:::MLPv0.5.0 resnet 1539716307.789943 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 6.3941814e-05}
-:::MLPv0.5.0 resnet 1539716338.8081341 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 9.5912721e-05}
-:::MLPv0.5.0 resnet 1539716370.605946541 (resnet_run_loop.py:481) input_size: 25600
-:::MLPv0.5.0 resnet 1539716370.606900454 (resnet_run_loop.py:495) eval_start
-:::MLPv0.5.0 resnet 1539716370.617015600 (official/resnet/imagenet_main.py:177) input_order
-:::MLPv0.5.0 resnet 1539716370.653812170 (imagenet_preprocessing.py:224) input_resize_aspect_preserving: {"min": 256}
-:::MLPv0.5.0 resnet 1539716370.674451590 (imagenet_preprocessing.py:289) input_resize: [224, 224]
-:::MLPv0.5.0 resnet 1539716370.682832003 (imagenet_preprocessing.py:134) input_central_crop: [224, 224]
-:::MLPv0.5.0 resnet 1539716370.694524765 (imagenet_preprocessing.py:173) input_mean_subtraction: [123.68, 116.78, 103.94]
-:::MLPv0.5.0 resnet 1539716370.718717337 (resnet_model.py:396) model_hp_initial_shape: [224, 224, 3]
-:::MLPv0.5.0 resnet 1539716370.734307051 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(3, 224, 224) -> (64, 112, 112)"
-:::MLPv0.5.0 resnet 1539716370.735153437 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716370.756954193 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 112, 112)"}
-:::MLPv0.5.0 resnet 1539716370.757780075 (resnet_model.py:417) model_hp_relu
-:::MLPv0.5.0 resnet 1539716370.761164188 (resnet_model.py:425) model_hp_initial_max_pool: "(64, 112, 112) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.762840509 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716370.763802290 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.777111053 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.778208017 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716370.779098034 (resnet_model.py:164) model_hp_projection_shortcut: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.800961494 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716370.813964128 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.815053701 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716370.837027550 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716370.837923288 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716370.850971699 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.851952314 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716370.873186111 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716370.874081135 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716370.887556791 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.888557911 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716370.909958839 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716370.910790205 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716370.912375450 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716370.914038181 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.914975643 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716370.915924311 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.928850412 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.929883480 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716370.962407112 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716370.963244438 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716370.976882458 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716370.978215694 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716370.999885798 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716371.000729799 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.013527393 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.014536381 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.036516428 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716371.037377834 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716371.039016962 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.040589094 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.041512966 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716371.042399883 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.054443598 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.055367470 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.077505827 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716371.078360081 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.091528654 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.092493057 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.114387274 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716371.115240335 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.129176617 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.130229712 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.151735306 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716371.152626514 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716371.154198885 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.155818462 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.157381773 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716371.158363819 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.172552824 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.173595667 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716371.174496651 (resnet_model.py:164) model_hp_projection_shortcut: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.196137428 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.208388090 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (128, 56, 56)"
-:::MLPv0.5.0 resnet 1539716371.209329605 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.231015682 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716371.231850147 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.247154236 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 56, 56) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.248196363 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716371.269721031 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.270572901 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.283307552 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.284232855 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.305608273 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.306540012 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716371.308152676 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.309848070 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.310852289 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716371.311810493 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.324080467 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.325070858 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.346801281 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.347671986 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.360682011 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.361728191 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.382938862 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.383775234 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.396614552 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.397608757 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.419632196 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.420474291 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716371.422101974 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.423712015 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.424610615 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716371.425484657 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.437613249 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.438634634 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.459997416 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.460868359 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.473891497 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.474915504 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.496156454 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.497026682 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.509894133 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.510844469 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.532710552 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.533583641 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716371.535134315 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.536683559 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.537604332 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716371.538563967 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.550745487 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.551673651 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.574620485 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.575479269 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.899750948 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.900853872 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.922526836 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.923406124 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.936459064 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.937478065 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716371.959039927 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716371.959884405 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716371.961618900 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716371.963212252 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.964890242 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716371.965884686 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716371.980025291 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716371.981097937 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716371.982061625 (resnet_model.py:164) model_hp_projection_shortcut: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.009623528 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.021925211 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (256, 28, 28)"
-:::MLPv0.5.0 resnet 1539716372.022954702 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.044744015 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716372.045590401 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.061194897 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 28, 28) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.062668562 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716372.086530924 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.087444782 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.100742579 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.101808786 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.130464554 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.131277084 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.132861853 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.134460211 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.135382652 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.136248350 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.149247169 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.150211096 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.171760559 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.172699928 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.185858011 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.186830282 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.208784580 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.209629297 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.223185301 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.224196434 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.252441406 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.253267288 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.254948139 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.256578922 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.257596731 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.258606672 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.271031141 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.272098303 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.293788433 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.294703245 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.307572365 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.308581591 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.330110788 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.330980062 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.343944550 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.344923019 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.372858524 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.373778105 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.375382900 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.377063751 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.378094196 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.378987074 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.391492844 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.392482042 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.414121151 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.414994955 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.429008245 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.429984570 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.451719522 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.452589512 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.468732834 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.470336676 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.503775597 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.504672527 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.506334543 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.507892370 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.508861780 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.509852171 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.522308826 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.523363352 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.545728445 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.546581507 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.559615612 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.560577631 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.582149267 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.583009481 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.595836401 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.596793890 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.625514030 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.626394749 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.628009081 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.629590511 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.630533695 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.631387234 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.643849611 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.644857168 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.666565180 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.667436600 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.680242777 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.681227446 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.702875614 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.703756571 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.716830492 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.717873335 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.745720148 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.746632099 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.748245239 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.749820709 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.751404285 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.752340317 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.767162323 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.768234491 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716372.769201756 (resnet_model.py:164) model_hp_projection_shortcut: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.797233582 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716372.809701920 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (512, 14, 14)"
-:::MLPv0.5.0 resnet 1539716372.810724497 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.832258701 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716372.833175898 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.847966671 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 14, 14) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.848985195 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716372.870685577 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716372.871538877 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.884602785 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.885567188 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.913537741 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716372.914429903 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716372.916043997 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.917570353 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.918609619 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716372.919517517 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.932535648 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.933475018 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.955540419 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716372.956398249 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716372.969494820 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716372.970468283 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716372.991911888 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716372.992809296 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716373.005783319 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.006736517 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716373.035068989 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716373.035985708 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716373.037735939 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716373.039408445 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.040402412 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716373.041257381 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.054044008 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.055001736 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716373.076649189 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716373.077678204 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716373.090589285 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.091554403 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716373.114003897 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716373.114868641 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716373.127604961 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.128547192 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716373.156557798 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716373.157408953 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716373.159055233 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716373.160750628 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716373.166387081 (resnet_model.py:455) model_hp_dense: 1001
-:::MLPv0.5.0 resnet 1539716373.185448885 (resnet_model.py:461) model_hp_final_shape: [1001]
-:::MLPv0.5.0 resnet 1539716373.197274923 (resnet_run_loop.py:249) model_hp_loss_fn: "categorical_cross_entropy"
-:::MLPv0.5.0 resnet 1539716373.229386330 (resnet_run_loop.py:264) model_exclude_bn_from_l2: true
-:::MLPv0.5.0 resnet 1539716373.230116606 (resnet_run_loop.py:268) model_l2_regularization: 0.0001
-:::MLPv0.5.0 resnet 1539716386.852867126 (resnet_run_loop.py:504) eval_stop
-:::MLPv0.5.0 resnet 1539716386.853560448 (resnet_run_loop.py:505) eval_size: 6400
-:::MLPv0.5.0 resnet 1539716386.854036093 (resnet_run_loop.py:506) eval_accuracy: 0.0012499999720603228
-:::MLPv0.5.0 resnet 1539716386.854490757 (resnet_run_loop.py:507) eval_target: 0.749
-:::MLPv0.5.0 resnet 1539716386.855098009 (resnet_run_loop.py:445) train_epoch: 4
-:::MLPv0.5.0 resnet 1539716386.855572224 (resnet_run_loop.py:445) train_epoch: 5
-:::MLPv0.5.0 resnet 1539716386.856010914 (resnet_run_loop.py:445) train_epoch: 6
-:::MLPv0.5.0 resnet 1539716386.856444359 (resnet_run_loop.py:445) train_epoch: 7
-:::MLPv0.5.0 resnet 1539716386.858455658 (official/resnet/imagenet_main.py:177) input_order
-:::MLPv0.5.0 resnet 1539716386.882511139 (resnet_run_loop.py:76) input_order
-:::MLPv0.5.0 resnet 1539716386.912208080 (imagenet_preprocessing.py:86) input_distorted_crop_min_object_covered: 0.1
-:::MLPv0.5.0 resnet 1539716386.913742781 (imagenet_preprocessing.py:88) input_distorted_crop_aspect_ratio_range: [0.75, 1.33]
-:::MLPv0.5.0 resnet 1539716386.915192366 (imagenet_preprocessing.py:90) input_distorted_crop_area_range: [0.05, 1.0]
-:::MLPv0.5.0 resnet 1539716386.916644812 (imagenet_preprocessing.py:92) input_distorted_crop_max_attempts: 100
-:::MLPv0.5.0 resnet 1539716386.924070835 (imagenet_preprocessing.py:114) input_random_flip
-:::MLPv0.5.0 resnet 1539716386.945967913 (imagenet_preprocessing.py:281) input_resize: [224, 224]
-:::MLPv0.5.0 resnet 1539716386.951725245 (imagenet_preprocessing.py:173) input_mean_subtraction: [123.68, 116.78, 103.94]
-:::MLPv0.5.0 resnet 1539716386.991346598 (resnet_model.py:396) model_hp_initial_shape: [224, 224, 3]
-:::MLPv0.5.0 resnet 1539716387.009206057 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(3, 224, 224) -> (64, 112, 112)"
-:::MLPv0.5.0 resnet 1539716387.010159016 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716387.055179596 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 112, 112)"}
-:::MLPv0.5.0 resnet 1539716387.056022644 (resnet_model.py:417) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.059495687 (resnet_model.py:425) model_hp_initial_max_pool: "(64, 112, 112) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.061192036 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716387.062260151 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.076822758 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.077916145 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.078895569 (resnet_model.py:164) model_hp_projection_shortcut: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.121352911 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.135078669 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.136046171 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.176357269 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.177221537 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.191541195 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.192517519 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.233788729 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.234698057 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.249076366 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.250067234 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.290715456 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.291623831 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716387.293307781 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.294997692 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.295937300 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716387.296908617 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.310950279 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.312014341 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.355982304 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.356941938 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.372096062 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.373092413 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.415580034 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.416480303 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.431459427 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.432438850 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.475224257 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.476146936 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716387.477910757 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.479665279 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.480600595 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716387.481593370 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.495878935 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.496885777 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.539625406 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.540536642 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.556501150 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.557507515 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716387.599439621 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716387.600450277 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716387.615603924 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716387.616592407 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.175709248 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716388.176688671 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716388.178497553 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.180250406 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716388.182025671 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716388.183063507 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716388.200723410 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.201810598 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716388.202839375 (resnet_model.py:164) model_hp_projection_shortcut: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.245965242 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.260187387 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (128, 56, 56)"
-:::MLPv0.5.0 resnet 1539716388.261157274 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.304519892 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716388.305392742 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.323521376 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 56, 56) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.324536324 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716388.368076086 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.368949413 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.383849144 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.385133982 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.428547621 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.429532528 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716388.431412935 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.433161020 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.434196711 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716388.435193062 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.450058937 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.451061249 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.495162487 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.496060848 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.511415243 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.512595654 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.556917429 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.557882071 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.573327541 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.574352980 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.618422270 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.619342089 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716388.621203899 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.623064280 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.624014378 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716388.624957800 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.641048431 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.642075300 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.685905218 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.686769724 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.702016830 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.702972412 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.746004105 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.746864319 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.761643887 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.762647152 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.806942701 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.807853460 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716388.809718370 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.811496496 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.812473774 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716388.813427925 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.827952147 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.828973532 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.872406006 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.873355389 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.888622761 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.889625788 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.934686422 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.935631275 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.951303959 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716388.952238798 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716388.994431257 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716388.995341301 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716388.997082233 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716388.998834133 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716389.000638962 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716389.001640320 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716389.018034220 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.019054174 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716389.020006180 (resnet_model.py:164) model_hp_projection_shortcut: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.070029497 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.084743977 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (256, 28, 28)"
-:::MLPv0.5.0 resnet 1539716389.085763931 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.128155231 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716389.129080057 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.146063805 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 28, 28) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.147046328 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716389.189240932 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.190173388 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.204809904 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.205743551 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.255081177 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.255977631 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716389.257701874 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.259369612 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.260384560 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716389.261298180 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.275278568 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.276218414 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.318720818 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.319700956 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.334418774 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.335405588 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.378194094 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.379118443 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.393964529 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.394981861 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.444052458 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.445066214 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716389.446829557 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.448704004 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.449802876 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716389.450815916 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.465457439 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.466541052 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.509909630 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.510807037 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.525568008 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.526677608 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.569640636 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.570514679 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.585612059 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.586668730 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.635984898 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.636857271 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716389.638623714 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.640360117 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.641310930 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716389.642203093 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.656053543 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.657108068 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.699182510 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.700091600 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.715093613 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.716064692 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.757806540 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.758713245 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.773760080 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.774826050 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716389.945610285 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716389.946630955 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716389.948271513 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716389.949923754 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.950803518 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716389.951653242 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.966622829 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716389.967632532 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.009719372 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.010600567 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.025610685 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.026608944 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.070242167 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.071240902 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.087631941 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.088659525 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.137356758 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.138340950 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716390.140078545 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.141789913 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.142753363 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716390.143775702 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.158147335 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.159131527 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.202203751 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.203105927 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.218373775 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.219363689 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.261928320 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.262821674 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.277991533 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.278991222 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.327921152 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.328836679 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716390.330540419 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.332322121 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.334002972 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716390.334942579 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.351259708 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.352214098 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716390.353119135 (resnet_model.py:164) model_hp_projection_shortcut: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.400824785 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.415027142 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (512, 14, 14)"
-:::MLPv0.5.0 resnet 1539716390.415998220 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.458482742 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716390.459419250 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.476434708 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 14, 14) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.477406979 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716390.520509243 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.521450758 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.536847115 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.537929773 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.586752176 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.587828398 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716390.589725018 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.591441154 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.592397213 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716390.593310356 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.607684135 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.608724356 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.651774883 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.652745247 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.668164015 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.669167280 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.711118221 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.712016344 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.727505684 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.728576660 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.777381897 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.778338432 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716390.780053854 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.781774998 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.782704115 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716390.783662558 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.797677994 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.798715353 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.840793133 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.841737509 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.856254816 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.857308865 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.898839951 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.899806976 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.914770842 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.915706635 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716390.963610649 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": true, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716390.964573383 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716390.966503859 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716390.968129396 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716390.973153114 (resnet_model.py:455) model_hp_dense: 1001
-:::MLPv0.5.0 resnet 1539716390.995549440 (resnet_model.py:461) model_hp_final_shape: [1001]
-:::MLPv0.5.0 resnet 1539716391.011470795 (resnet_run_loop.py:249) model_hp_loss_fn: "categorical_cross_entropy"
-:::MLPv0.5.0 resnet 1539716391.046864271 (resnet_run_loop.py:264) model_exclude_bn_from_l2: true
-:::MLPv0.5.0 resnet 1539716391.047663689 (resnet_run_loop.py:268) model_l2_regularization: 0.0001
-:::MLPv0.5.0 resnet 1539716391.247353077 (resnet_run_loop.py:290) opt_name: "stochastic_gradient_descent_with_momentum"
-:::MLPv0.5.0 resnet 1539716391.248133183 (resnet_run_loop.py:291) opt_momentum: 0.9
-:::MLPv0.5.0 resnet 1539716404.7773719 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 0.000127883628}
-:::MLPv0.5.0 resnet 1539716439.1605461 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 0.000159854535}
-:::MLPv0.5.0 resnet 1539716470.1754661 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 0.000191825442}
-:::MLPv0.5.0 resnet 1539716501.080837 (resnet_run_loop.py:281) opt_learning_rate: {"deferred": true, "value": 0.000223796349}
-:::MLPv0.5.0 resnet 1539716533.030520916 (resnet_run_loop.py:481) input_size: 25600
-:::MLPv0.5.0 resnet 1539716533.030993223 (resnet_run_loop.py:495) eval_start
-:::MLPv0.5.0 resnet 1539716533.038758993 (official/resnet/imagenet_main.py:177) input_order
-:::MLPv0.5.0 resnet 1539716533.072755098 (imagenet_preprocessing.py:224) input_resize_aspect_preserving: {"min": 256}
-:::MLPv0.5.0 resnet 1539716533.092728853 (imagenet_preprocessing.py:289) input_resize: [224, 224]
-:::MLPv0.5.0 resnet 1539716533.101056337 (imagenet_preprocessing.py:134) input_central_crop: [224, 224]
-:::MLPv0.5.0 resnet 1539716533.111096859 (imagenet_preprocessing.py:173) input_mean_subtraction: [123.68, 116.78, 103.94]
-:::MLPv0.5.0 resnet 1539716533.133472443 (resnet_model.py:396) model_hp_initial_shape: [224, 224, 3]
-:::MLPv0.5.0 resnet 1539716533.148386478 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(3, 224, 224) -> (64, 112, 112)"
-:::MLPv0.5.0 resnet 1539716533.149225473 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716533.170512915 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 112, 112)"}
-:::MLPv0.5.0 resnet 1539716533.171267509 (resnet_model.py:417) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.174275637 (resnet_model.py:425) model_hp_initial_max_pool: "(64, 112, 112) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.175843000 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.176809311 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.190997601 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.191988230 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.192893267 (resnet_model.py:164) model_hp_projection_shortcut: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.213391304 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.225266218 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.226243258 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.246613026 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.247494936 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.259941816 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.260900974 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.281647444 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.282531023 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.295063734 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.296067476 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.317403078 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.318254471 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716533.319886684 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.321635008 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.322646141 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.323613882 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.336032867 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.337027788 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.357959270 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.358818531 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.371611357 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.372562885 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.394973278 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.395844936 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.408946276 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.409979582 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.432271719 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.433211088 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716533.434962749 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.436665773 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.437682629 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.438593388 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.451451302 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.452408552 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.473956585 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.474872351 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.487658978 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (64, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.488699198 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 64, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.509781837 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(64, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.510666847 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.523177624 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(64, 56, 56) -> (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.524127245 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.544886589 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.545775414 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716533.547355890 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.548913002 (resnet_model.py:195) model_hp_end_block: " Block Output: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.550476789 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.551438808 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (256, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.565510273 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.566509724 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716533.567413807 (resnet_model.py:164) model_hp_projection_shortcut: "(256, 56, 56) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.587878942 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.599695683 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 56, 56) -> (128, 56, 56)"
-:::MLPv0.5.0 resnet 1539716533.600656509 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.621340275 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 56, 56)"}
-:::MLPv0.5.0 resnet 1539716533.622181892 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.636133671 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 56, 56) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.637078285 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716533.657693863 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.658554792 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.670963049 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.671915054 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.692467928 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.693314791 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716533.694906473 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.696492195 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.697396517 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.698275805 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.710075140 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.711021662 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.732477188 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.733333349 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.745937347 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.746889591 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.767845631 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.768707514 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.781164885 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.782133818 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.802746296 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.803594351 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716533.805142879 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.806746244 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.807648420 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.808539629 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.820353508 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.821264982 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.842029333 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.842902660 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.855397701 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.856356382 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.877576828 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.878453255 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.891606808 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.892558098 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.913204908 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.914083481 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716533.915666103 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.917202950 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.918130636 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716533.919022083 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.930858612 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.931811094 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.952532053 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.953384399 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716533.965845585 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (128, 28, 28)"
-:::MLPv0.5.0 resnet 1539716533.966796398 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 128, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716533.987557173 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(128, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716533.988448620 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.000908375 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(128, 28, 28) -> (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716534.001882792 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.022808313 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716534.023656130 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.025191784 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.026788473 (resnet_model.py:195) model_hp_end_block: " Block Output: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716534.028337955 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.029255629 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (512, 28, 28)"
-:::MLPv0.5.0 resnet 1539716534.042987823 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.043973684 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716534.044873476 (resnet_model.py:164) model_hp_projection_shortcut: "(512, 28, 28) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.073022366 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.085116386 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 28, 28) -> (256, 28, 28)"
-:::MLPv0.5.0 resnet 1539716534.086097002 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.107146740 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 28, 28)"}
-:::MLPv0.5.0 resnet 1539716534.108045101 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.122306108 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 28, 28) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.123228312 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716534.144118786 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.144978523 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.157525778 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.158510208 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.185557842 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.186441422 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.187998772 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.189582586 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.190506220 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.191387415 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.203381777 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.204324007 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.225798845 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.226664782 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.239253521 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.240214348 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.261229992 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.262105703 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.274773121 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.275763750 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.302831650 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.303704500 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.305266619 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.306884289 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.307811022 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.308711052 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.320757627 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.321734905 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.342823505 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.343689919 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.356308699 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.357239008 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.378148794 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.379006386 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.392617464 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.393560410 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.420948267 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.421855927 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.423449039 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.425010681 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.425954103 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.426848173 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.438972950 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.439935446 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.460889816 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.461838961 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.474670887 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.475614786 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.496570110 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.497422934 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.510802269 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.512322426 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.539538622 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.540396690 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.541992426 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.543601274 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.544508696 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.545392990 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.558122396 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.559071064 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.580108881 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.580963373 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.593628168 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.594600201 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.615621328 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.616482258 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.629108429 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.630082130 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.657198668 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.658095598 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.659713984 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.661275625 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.662217140 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.663110733 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.675417662 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.676382303 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.697387457 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.698244810 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.710992575 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (256, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.711945295 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 256, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.733761787 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(256, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.734616518 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.747339964 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(256, 14, 14) -> (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.748257875 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 1024, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.775506973 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(1024, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.776372194 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716534.777974844 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.779561281 (resnet_model.py:195) model_hp_end_block: " Block Output: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.781091928 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716534.782075644 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (1024, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.795804024 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716534.796792746 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716534.797716141 (resnet_model.py:164) model_hp_projection_shortcut: "(1024, 14, 14) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716534.824719191 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716534.836930513 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(1024, 14, 14) -> (512, 14, 14)"
-:::MLPv0.5.0 resnet 1539716534.837911367 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716534.858847618 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 14, 14)"}
-:::MLPv0.5.0 resnet 1539716534.859707594 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716534.874719858 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 14, 14) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716534.875684500 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 2}
-:::MLPv0.5.0 resnet 1539716535.131258488 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.132164001 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.144984961 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.145958424 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.173089504 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.173974037 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716535.175588608 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.177130222 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.178061724 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716535.178951740 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.191032887 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.191985846 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.212948084 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.213841200 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.226536274 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.227494240 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.248571873 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.249439240 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.262138367 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.263087511 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.290819168 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.291680098 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716535.293234110 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.294844627 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.295750380 (resnet_model.py:157) model_hp_begin_block: {"block_type": "bottleneck_block"}
-:::MLPv0.5.0 resnet 1539716535.296639204 (resnet_model.py:157) model_hp_resnet_topology: " Block Input: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.308673382 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(2048, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.309630156 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.330932379 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.331795931 (resnet_model.py:173) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.344495535 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (512, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.345446110 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 512, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.366499424 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(512, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.367374420 (resnet_model.py:181) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.380116463 (resnet_model.py:112) model_hp_conv2d_fixed_padding: "(512, 7, 7) -> (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.381062746 (resnet_model.py:112) model_hp_conv2d_fixed_padding: {"use_bias": false, "filters": 2048, "initializer": "truncated_normal", "stride": 1}
-:::MLPv0.5.0 resnet 1539716535.408458948 (resnet_model.py:62) model_hp_batch_norm: {"scale": true, "epsilon": 1e-05, "center": true, "training": false, "momentum": 0.9, "shape": "(2048, 7, 7)"}
-:::MLPv0.5.0 resnet 1539716535.409321547 (resnet_model.py:189) model_hp_shorcut_add
-:::MLPv0.5.0 resnet 1539716535.410909414 (resnet_model.py:192) model_hp_relu
-:::MLPv0.5.0 resnet 1539716535.412492752 (resnet_model.py:195) model_hp_end_block: " Block Output: (2048, 7, 7)"
-:::MLPv0.5.0 resnet 1539716535.417588472 (resnet_model.py:455) model_hp_dense: 1001
-:::MLPv0.5.0 resnet 1539716535.436456919 (resnet_model.py:461) model_hp_final_shape: [1001]
-:::MLPv0.5.0 resnet 1539716535.447912931 (resnet_run_loop.py:249) model_hp_loss_fn: "categorical_cross_entropy"
-:::MLPv0.5.0 resnet 1539716535.479585886 (resnet_run_loop.py:264) model_exclude_bn_from_l2: true
-:::MLPv0.5.0 resnet 1539716535.480349064 (resnet_run_loop.py:268) model_l2_regularization: 0.0001
-:::MLPv0.5.0 resnet 1539716548.641486406 (resnet_run_loop.py:504) eval_stop
-:::MLPv0.5.0 resnet 1539716548.642030001 (resnet_run_loop.py:505) eval_size: 6400
-:::MLPv0.5.0 resnet 1539716548.642558575 (resnet_run_loop.py:506) eval_accuracy: 0.0014062500558793545
-:::MLPv0.5.0 resnet 1539716548.643030167 (resnet_run_loop.py:507) eval_target: 0.749
-:::MLPv0.5.0 resnet 1539716548.643552780 (resnet_run_loop.py:518) run_stop: {"success": false}
-:::MLPv0.5.0 resnet 1539716548.644066334 (resnet_run_loop.py:519) run_final
diff --git a/compliance/example_logs/ssd_example.log b/compliance/example_logs/ssd_example.log
deleted file mode 100644
index 300cf888b..000000000
--- a/compliance/example_logs/ssd_example.log
+++ /dev/null
@@ -1,103 +0,0 @@
-:::MLPv0.5.0 ssd 1540247495.296750784 (train.py:277) run_start
-:::MLPv0.5.0 ssd 1540247495.484981537 (train.py:52) feature_sizes: [38, 19, 10, 5, 3, 1]
-:::MLPv0.5.0 ssd 1540247495.485468149 (train.py:55) steps: [8, 16, 32, 64, 100, 300]
-:::MLPv0.5.0 ssd 1540247495.485914469 (train.py:59) scales: [21, 45, 99, 153, 207, 261, 315]
-:::MLPv0.5.0 ssd 1540247495.486357212 (train.py:62) aspect_ratios: [[2], [2, 3], [2, 3], [2, 3], [2], [2]]
-:::MLPv0.5.0 ssd 1540247495.492438316 (train.py:66) num_default_boxes: 8732
-:::MLPv0.5.0 ssd 1540247495.492895126 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:307) num_cropping_iterations: 1
-:::MLPv0.5.0 ssd 1540247495.493321180 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:430) random_flip_probability: 0.5
-:::MLPv0.5.0 ssd 1540247495.493689537 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:475) data_normalization_mean: [0.485, 0.456, 0.406]
-:::MLPv0.5.0 ssd 1540247495.494050026 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:477) data_normalization_std: [0.229, 0.224, 0.225]
-:::MLPv0.5.0 ssd 1540247495.494496346 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:307) num_cropping_iterations: 1
-:::MLPv0.5.0 ssd 1540247495.494894266 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:430) random_flip_probability: 0.5
-:::MLPv0.5.0 ssd 1540247495.495255232 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:475) data_normalization_mean: [0.485, 0.456, 0.406]
-:::MLPv0.5.0 ssd 1540247495.495614767 (/home/n/GitHub/training/single_stage_detector/ssd/utils.py:477) data_normalization_std: [0.229, 0.224, 0.225]
-:::MLPv0.5.0 ssd 1540247495.495964527 (train.py:156) input_size: 300
-:::MLPv0.5.0 ssd 1540247507.248809814 (train.py:170) input_shard
-:::MLPv0.5.0 ssd 1540247507.249204636 (train.py:171) input_order
-:::MLPv0.5.0 ssd 1540247507.249577284 (train.py:172) input_batch_size: 32
-:::MLPv0.5.0 ssd 1540247507.579035759 (/home/n/GitHub/training/single_stage_detector/ssd/ssd300.py:22) backbone: "resnet34"
-:::MLPv0.5.0 ssd 1540247507.579440355 (/home/n/GitHub/training/single_stage_detector/ssd/ssd300.py:27) loc_conf_out_channels: [256, 512, 512, 256, 256, 256]
-:::MLPv0.5.0 ssd 1540247507.596049309 (/home/n/GitHub/training/single_stage_detector/ssd/ssd300.py:39) num_defaults_per_cell: [4, 6, 6, 6, 4, 4]
-:::MLPv0.5.0 ssd 1540247508.969627857 (train.py:193) opt_name: "SGD"
-:::MLPv0.5.0 ssd 1540247508.970012665 (train.py:194) opt_learning_rate: 0.001
-:::MLPv0.5.0 ssd 1540247508.970359087 (train.py:195) opt_momentum: 0.9
-:::MLPv0.5.0 ssd 1540247508.970699310 (train.py:197) opt_weight_decay: 0.0005
-:::MLPv0.5.0 ssd 1540247508.971044302 (train.py:205) train_loop
-:::MLPv0.5.0 ssd 1540247508.971385717 (train.py:207) train_epoch: 0
-:::MLPv0.5.0 ssd 1540248445.739219666 (train.py:207) train_epoch: 1
-:::MLPv0.5.0 ssd 1540249385.273350477 (train.py:207) train_epoch: 2
-:::MLPv0.5.0 ssd 1540250321.425246000 (train.py:207) train_epoch: 3
-:::MLPv0.5.0 ssd 1540251258.187723875 (train.py:207) train_epoch: 4
-:::MLPv0.5.0 ssd 1540252194.437596321 (train.py:207) train_epoch: 5
-:::MLPv0.5.0 ssd 1540253130.675116301 (train.py:207) train_epoch: 6
-:::MLPv0.5.0 ssd 1540254067.046112537 (train.py:207) train_epoch: 7
-:::MLPv0.5.0 ssd 1540255002.359492779 (train.py:207) train_epoch: 8
-:::MLPv0.5.0 ssd 1540255937.986558199 (train.py:207) train_epoch: 9
-:::MLPv0.5.0 ssd 1540256873.417140484 (train.py:207) train_epoch: 10
-:::MLPv0.5.0 ssd 1540257808.319952726 (train.py:207) train_epoch: 11
-:::MLPv0.5.0 ssd 1540258743.776263475 (train.py:207) train_epoch: 12
-:::MLPv0.5.0 ssd 1540259679.752927542 (train.py:207) train_epoch: 13
-:::MLPv0.5.0 ssd 1540260615.683235645 (train.py:207) train_epoch: 14
-:::MLPv0.5.0 ssd 1540261552.390225649 (train.py:207) train_epoch: 15
-:::MLPv0.5.0 ssd 1540262489.468440056 (train.py:207) train_epoch: 16
-:::MLPv0.5.0 ssd 1540263425.589217424 (train.py:207) train_epoch: 17
-:::MLPv0.5.0 ssd 1540264361.546042204 (train.py:207) train_epoch: 18
-:::MLPv0.5.0 ssd 1540265297.222844839 (train.py:207) train_epoch: 19
-:::MLPv0.5.0 ssd 1540266233.569544554 (train.py:207) train_epoch: 20
-:::MLPv0.5.0 ssd 1540267169.144786119 (train.py:207) train_epoch: 21
-:::MLPv0.5.0 ssd 1540268105.018629551 (train.py:207) train_epoch: 22
-:::MLPv0.5.0 ssd 1540269041.111955166 (train.py:207) train_epoch: 23
-:::MLPv0.5.0 ssd 1540269977.069438934 (train.py:207) train_epoch: 24
-:::MLPv0.5.0 ssd 1540270912.911368132 (train.py:207) train_epoch: 25
-:::MLPv0.5.0 ssd 1540271848.584961176 (train.py:207) train_epoch: 26
-:::MLPv0.5.0 ssd 1540272785.012038708 (train.py:207) train_epoch: 27
-:::MLPv0.5.0 ssd 1540273721.774573088 (train.py:207) train_epoch: 28
-:::MLPv0.5.0 ssd 1540274658.045384407 (train.py:207) train_epoch: 29
-:::MLPv0.5.0 ssd 1540275593.669971943 (train.py:207) train_epoch: 30
-:::MLPv0.5.0 ssd 1540276529.381466866 (train.py:207) train_epoch: 31
-:::MLPv0.5.0 ssd 1540277465.441329241 (train.py:207) train_epoch: 32
-:::MLPv0.5.0 ssd 1540278159.841326237 (train.py:82) nms_threshold: 0.5
-:::MLPv0.5.0 ssd 1540278159.841703176 (train.py:84) nms_max_detections: 200
-:::MLPv0.5.0 ssd 1540278159.841991186 (train.py:86) eval_start: 32
-:::MLPv0.5.0 ssd 1540278624.957715034 (train.py:133) eval_size: 4952
-:::MLPv0.5.0 ssd 1540278624.958045244 (train.py:136) eval_accuracy: {"epoch": 32, "value": 0.1767261512048043}
-:::MLPv0.5.0 ssd 1540278624.958331823 (train.py:139) eval_iteration_accuracy: {"iteration": 120000, "value": 0.1767261512048043}
-:::MLPv0.5.0 ssd 1540278624.958611488 (train.py:140) eval_target: 0.212
-:::MLPv0.5.0 ssd 1540278624.958888054 (train.py:141) eval_stop: 32
-:::MLPv0.5.0 ssd 1540278859.034478188 (train.py:207) train_epoch: 33
-:::MLPv0.5.0 ssd 1540279776.504349232 (train.py:207) train_epoch: 34
-:::MLPv0.5.0 ssd 1540280694.871150732 (train.py:207) train_epoch: 35
-:::MLPv0.5.0 ssd 1540281612.904750109 (train.py:207) train_epoch: 36
-:::MLPv0.5.0 ssd 1540282530.138573885 (train.py:207) train_epoch: 37
-:::MLPv0.5.0 ssd 1540283448.168450356 (train.py:207) train_epoch: 38
-:::MLPv0.5.0 ssd 1540284366.526405096 (train.py:207) train_epoch: 39
-:::MLPv0.5.0 ssd 1540285283.962305069 (train.py:207) train_epoch: 40
-:::MLPv0.5.0 ssd 1540286201.935405970 (train.py:207) train_epoch: 41
-:::MLPv0.5.0 ssd 1540287120.724055529 (train.py:207) train_epoch: 42
-:::MLPv0.5.0 ssd 1540288037.873018026 (train.py:207) train_epoch: 43
-:::MLPv0.5.0 ssd 1540288640.754241705 (train.py:217) opt_learning_rate: 0.0001
-:::MLPv0.5.0 ssd 1540288641.213376760 (train.py:82) nms_threshold: 0.5
-:::MLPv0.5.0 ssd 1540288641.213843822 (train.py:84) nms_max_detections: 200
-:::MLPv0.5.0 ssd 1540288641.214268446 (train.py:86) eval_start: 43
-:::MLPv0.5.0 ssd 1540289130.900609016 (train.py:133) eval_size: 4952
-:::MLPv0.5.0 ssd 1540289130.900934935 (train.py:136) eval_accuracy: {"epoch": 43, "value": 0.18708427961724622}
-:::MLPv0.5.0 ssd 1540289130.901222467 (train.py:139) eval_iteration_accuracy: {"iteration": 160000, "value": 0.18708427961724622}
-:::MLPv0.5.0 ssd 1540289130.901501417 (train.py:140) eval_target: 0.212
-:::MLPv0.5.0 ssd 1540289130.901779413 (train.py:141) eval_stop: 43
-:::MLPv0.5.0 ssd 1540289443.537181616 (train.py:207) train_epoch: 44
-:::MLPv0.5.0 ssd 1540290361.537575960 (train.py:207) train_epoch: 45
-:::MLPv0.5.0 ssd 1540291280.019073248 (train.py:207) train_epoch: 46
-:::MLPv0.5.0 ssd 1540292198.773092508 (train.py:207) train_epoch: 47
-:::MLPv0.5.0 ssd 1540293115.953065872 (train.py:207) train_epoch: 48
-:::MLPv0.5.0 ssd 1540294034.054209709 (train.py:207) train_epoch: 49
-:::MLPv0.5.0 ssd 1540294139.414384604 (train.py:82) nms_threshold: 0.5
-:::MLPv0.5.0 ssd 1540294139.414769411 (train.py:84) nms_max_detections: 200
-:::MLPv0.5.0 ssd 1540294139.415052652 (train.py:86) eval_start: 49
-:::MLPv0.5.0 ssd 1540294574.994000435 (train.py:133) eval_size: 4952
-:::MLPv0.5.0 ssd 1540294574.994378567 (train.py:136) eval_accuracy: {"epoch": 49, "value": 0.2228821952756952}
-:::MLPv0.5.0 ssd 1540294574.994705200 (train.py:139) eval_iteration_accuracy: {"iteration": 180000, "value": 0.2228821952756952}
-:::MLPv0.5.0 ssd 1540294574.995025396 (train.py:140) eval_target: 0.212
-:::MLPv0.5.0 ssd 1540294574.995343685 (train.py:141) eval_stop: 49
-:::MLPv0.5.0 ssd 1540294575.562962294 (train.py:282) run_stop: {"success": true}
-:::MLPv0.5.0 ssd 1540294575.563240528 (train.py:283) run_final
diff --git a/compliance/example_submission/code/ncf/README.md b/compliance/example_submission/code/ncf/README.md
deleted file mode 100644
index 53d8d9de2..000000000
--- a/compliance/example_submission/code/ncf/README.md
+++ /dev/null
@@ -1,1 +0,0 @@
-Test README.md
diff --git a/compliance/example_submission/code/ncf/preproc_dataset.sh b/compliance/example_submission/code/ncf/preproc_dataset.sh
deleted file mode 100644
index 96da985ed..000000000
--- a/compliance/example_submission/code/ncf/preproc_dataset.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test preproc_dataset.sh
diff --git a/compliance/example_submission/code/ncf/run_and_time_zpu1x.sh b/compliance/example_submission/code/ncf/run_and_time_zpu1x.sh
deleted file mode 100644
index ce89ebf90..000000000
--- a/compliance/example_submission/code/ncf/run_and_time_zpu1x.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test run_and_time_zpu1x.sh
diff --git a/compliance/example_submission/code/ncf/setup_zpu1x.sh b/compliance/example_submission/code/ncf/setup_zpu1x.sh
deleted file mode 100644
index 761cfd88e..000000000
--- a/compliance/example_submission/code/ncf/setup_zpu1x.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test setup_zpu1x.sh
diff --git a/compliance/example_submission/code/resnet/README.md b/compliance/example_submission/code/resnet/README.md
deleted file mode 100644
index 53d8d9de2..000000000
--- a/compliance/example_submission/code/resnet/README.md
+++ /dev/null
@@ -1,1 +0,0 @@
-Test README.md
diff --git a/compliance/example_submission/code/resnet/preproc_dataset.sh b/compliance/example_submission/code/resnet/preproc_dataset.sh
deleted file mode 100644
index 96da985ed..000000000
--- a/compliance/example_submission/code/resnet/preproc_dataset.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test preproc_dataset.sh
diff --git a/compliance/example_submission/code/resnet/run_and_time_zpu1x.sh b/compliance/example_submission/code/resnet/run_and_time_zpu1x.sh
deleted file mode 100644
index ce89ebf90..000000000
--- a/compliance/example_submission/code/resnet/run_and_time_zpu1x.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test run_and_time_zpu1x.sh
diff --git a/compliance/example_submission/code/resnet/run_and_time_zpu8x.sh b/compliance/example_submission/code/resnet/run_and_time_zpu8x.sh
deleted file mode 100644
index 18d9a5771..000000000
--- a/compliance/example_submission/code/resnet/run_and_time_zpu8x.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test run_and_time_z100x8.sh
diff --git a/compliance/example_submission/code/resnet/setup_zpu1x.sh b/compliance/example_submission/code/resnet/setup_zpu1x.sh
deleted file mode 100644
index 761cfd88e..000000000
--- a/compliance/example_submission/code/resnet/setup_zpu1x.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test setup_zpu1x.sh
diff --git a/compliance/example_submission/code/resnet/setup_zpu8x.sh b/compliance/example_submission/code/resnet/setup_zpu8x.sh
deleted file mode 100644
index 705b542ab..000000000
--- a/compliance/example_submission/code/resnet/setup_zpu8x.sh
+++ /dev/null
@@ -1,1 +0,0 @@
-Test setup_zpux8.sh
diff --git a/compliance/example_submission/results/zpu1x/entry.json b/compliance/example_submission/results/zpu1x/entry.json
deleted file mode 100644
index 377b5406d..000000000
--- a/compliance/example_submission/results/zpu1x/entry.json
+++ /dev/null
@@ -1,29 +0,0 @@
-{
-    "division": "open",    
-    "status": "on-premise",    
-    "hardware": "1 ZPU",    
-    "framework": "TensorFlow 1.12",    
-    "power": "https://mlperf.org/",    
-    "notes": "ZPUs are the best",    
-    "interconnect": "InfiniBand ring",    
-    "os": "Ubuntu 18.10",    
-    "libraries": "cuDNN 7.3",    
-    "compilers": "N/A",    
-    "nodes": [
-            {
-                "num_nodes": 1,
-                "cpu": "Skylake X",
-                "num_cores": 18,
-                "num_vcpus": 0,
-                "accelerator": "zpu",
-                "num_accelerators": 1,
-                "sys_mem_size": 64,
-                "sys_storage_type": "SSD",
-                "sys_storage_size": 512,
-                "cpu_accel_interconnect": "zpu-intercon",
-                "network_card": "100 Gb InfiniBand",
-                "num_network_cards": 1,
-                "notes": "this is not a real product" 
-            }
-    ]
-}
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_0.txt b/compliance/example_submission/results/zpu1x/ncf/result_0.txt
deleted file mode 100644
index 209e3ef4b..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_0.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-20
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_1.txt b/compliance/example_submission/results/zpu1x/ncf/result_1.txt
deleted file mode 100644
index aabe6ec39..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_1.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-21
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_10.txt b/compliance/example_submission/results/zpu1x/ncf/result_10.txt
deleted file mode 100644
index 64bb6b746..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_10.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-30
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_11.txt b/compliance/example_submission/results/zpu1x/ncf/result_11.txt
deleted file mode 100644
index e85087aff..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_11.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-31
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_12.txt b/compliance/example_submission/results/zpu1x/ncf/result_12.txt
deleted file mode 100644
index f5c89552b..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_12.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-32
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_13.txt b/compliance/example_submission/results/zpu1x/ncf/result_13.txt
deleted file mode 100644
index bb95160cb..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_13.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-33
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_14.txt b/compliance/example_submission/results/zpu1x/ncf/result_14.txt
deleted file mode 100644
index a78736459..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_14.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-34
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_15.txt b/compliance/example_submission/results/zpu1x/ncf/result_15.txt
deleted file mode 100644
index 8f92bfdd4..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_15.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-35
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_16.txt b/compliance/example_submission/results/zpu1x/ncf/result_16.txt
deleted file mode 100644
index 7facc8993..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_16.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-36
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_17.txt b/compliance/example_submission/results/zpu1x/ncf/result_17.txt
deleted file mode 100644
index 81b5c5d06..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_17.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-37
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_18.txt b/compliance/example_submission/results/zpu1x/ncf/result_18.txt
deleted file mode 100644
index e522732c7..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_18.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-38
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_19.txt b/compliance/example_submission/results/zpu1x/ncf/result_19.txt
deleted file mode 100644
index a2720097d..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_19.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-39
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_2.txt b/compliance/example_submission/results/zpu1x/ncf/result_2.txt
deleted file mode 100644
index 2bd5a0a98..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_2.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-22
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_20.txt b/compliance/example_submission/results/zpu1x/ncf/result_20.txt
deleted file mode 100644
index 425151f3a..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_20.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-40
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_21.txt b/compliance/example_submission/results/zpu1x/ncf/result_21.txt
deleted file mode 100644
index 87523dd7a..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_21.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-41
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_22.txt b/compliance/example_submission/results/zpu1x/ncf/result_22.txt
deleted file mode 100644
index d81cc0710..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_22.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-42
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_23.txt b/compliance/example_submission/results/zpu1x/ncf/result_23.txt
deleted file mode 100644
index 920a13966..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_23.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-43
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_24.txt b/compliance/example_submission/results/zpu1x/ncf/result_24.txt
deleted file mode 100644
index c739b42c4..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_24.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-44
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_25.txt b/compliance/example_submission/results/zpu1x/ncf/result_25.txt
deleted file mode 100644
index ea90ee319..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_25.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-45
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_26.txt b/compliance/example_submission/results/zpu1x/ncf/result_26.txt
deleted file mode 100644
index 9e5feb525..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_26.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-46
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_27.txt b/compliance/example_submission/results/zpu1x/ncf/result_27.txt
deleted file mode 100644
index abac1ea7b..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_27.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-47
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_28.txt b/compliance/example_submission/results/zpu1x/ncf/result_28.txt
deleted file mode 100644
index 21e72e8ac..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_28.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-48
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_29.txt b/compliance/example_submission/results/zpu1x/ncf/result_29.txt
deleted file mode 100644
index 95f9650f0..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_29.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-49
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_3.txt b/compliance/example_submission/results/zpu1x/ncf/result_3.txt
deleted file mode 100644
index 409940768..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_3.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-23
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_30.txt b/compliance/example_submission/results/zpu1x/ncf/result_30.txt
deleted file mode 100644
index e373ee695..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_30.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-50
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_31.txt b/compliance/example_submission/results/zpu1x/ncf/result_31.txt
deleted file mode 100644
index 82cced27d..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_31.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-51
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_32.txt b/compliance/example_submission/results/zpu1x/ncf/result_32.txt
deleted file mode 100644
index 0691f67b2..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_32.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-52
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_33.txt b/compliance/example_submission/results/zpu1x/ncf/result_33.txt
deleted file mode 100644
index 59343b09e..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_33.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-53
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_34.txt b/compliance/example_submission/results/zpu1x/ncf/result_34.txt
deleted file mode 100644
index fb1e7bc86..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_34.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-54
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_35.txt b/compliance/example_submission/results/zpu1x/ncf/result_35.txt
deleted file mode 100644
index c3f407c09..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_35.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-55
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_36.txt b/compliance/example_submission/results/zpu1x/ncf/result_36.txt
deleted file mode 100644
index f6b91e0e1..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_36.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-56
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_37.txt b/compliance/example_submission/results/zpu1x/ncf/result_37.txt
deleted file mode 100644
index e1617e842..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_37.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-57
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_38.txt b/compliance/example_submission/results/zpu1x/ncf/result_38.txt
deleted file mode 100644
index 8c61d23e1..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_38.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-58
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_39.txt b/compliance/example_submission/results/zpu1x/ncf/result_39.txt
deleted file mode 100644
index 95f9650f0..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_39.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-49
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_4.txt b/compliance/example_submission/results/zpu1x/ncf/result_4.txt
deleted file mode 100644
index a45fd52cc..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_4.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-24
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_40.txt b/compliance/example_submission/results/zpu1x/ncf/result_40.txt
deleted file mode 100644
index abdfb053e..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_40.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-60
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_41.txt b/compliance/example_submission/results/zpu1x/ncf/result_41.txt
deleted file mode 100644
index b1e7d265f..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_41.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-61
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_42.txt b/compliance/example_submission/results/zpu1x/ncf/result_42.txt
deleted file mode 100644
index a8fa06e1b..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_42.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-62
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_43.txt b/compliance/example_submission/results/zpu1x/ncf/result_43.txt
deleted file mode 100644
index 4b9026d8e..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_43.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-63
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_44.txt b/compliance/example_submission/results/zpu1x/ncf/result_44.txt
deleted file mode 100644
index 900731ffd..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_44.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-64
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_45.txt b/compliance/example_submission/results/zpu1x/ncf/result_45.txt
deleted file mode 100644
index 1479e19b5..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_45.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-65
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_46.txt b/compliance/example_submission/results/zpu1x/ncf/result_46.txt
deleted file mode 100644
index 69a893aa3..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_46.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-66
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_47.txt b/compliance/example_submission/results/zpu1x/ncf/result_47.txt
deleted file mode 100644
index 3fdcd7c44..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_47.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-67
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_48.txt b/compliance/example_submission/results/zpu1x/ncf/result_48.txt
deleted file mode 100644
index 38b10c1b2..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_48.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-68
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_49.txt b/compliance/example_submission/results/zpu1x/ncf/result_49.txt
deleted file mode 100644
index b5489e5e5..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_49.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-69
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_5.txt b/compliance/example_submission/results/zpu1x/ncf/result_5.txt
deleted file mode 100644
index 7273c0fa8..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_5.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-25
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_50.txt b/compliance/example_submission/results/zpu1x/ncf/result_50.txt
deleted file mode 100644
index 2bbd69c2e..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_50.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-70
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_51.txt b/compliance/example_submission/results/zpu1x/ncf/result_51.txt
deleted file mode 100644
index 39f5b6931..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_51.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-71
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_52.txt b/compliance/example_submission/results/zpu1x/ncf/result_52.txt
deleted file mode 100644
index ea70ce013..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_52.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-72
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_53.txt b/compliance/example_submission/results/zpu1x/ncf/result_53.txt
deleted file mode 100644
index d69c74c8b..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_53.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-73
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_54.txt b/compliance/example_submission/results/zpu1x/ncf/result_54.txt
deleted file mode 100644
index fff0a2476..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_54.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-74
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_55.txt b/compliance/example_submission/results/zpu1x/ncf/result_55.txt
deleted file mode 100644
index 78eb67cee..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_55.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-75
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_56.txt b/compliance/example_submission/results/zpu1x/ncf/result_56.txt
deleted file mode 100644
index dd475631b..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_56.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-76
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_57.txt b/compliance/example_submission/results/zpu1x/ncf/result_57.txt
deleted file mode 100644
index 987e7ca9a..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_57.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-77
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_58.txt b/compliance/example_submission/results/zpu1x/ncf/result_58.txt
deleted file mode 100644
index 8e14edce9..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_58.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-78
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_59.txt b/compliance/example_submission/results/zpu1x/ncf/result_59.txt
deleted file mode 100644
index 85322d0b5..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_59.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-79
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_6.txt b/compliance/example_submission/results/zpu1x/ncf/result_6.txt
deleted file mode 100644
index 6f4247a62..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_6.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-26
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_60.txt b/compliance/example_submission/results/zpu1x/ncf/result_60.txt
deleted file mode 100644
index d15a2cc44..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_60.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-80
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_61.txt b/compliance/example_submission/results/zpu1x/ncf/result_61.txt
deleted file mode 100644
index d88e31369..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_61.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-81
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_62.txt b/compliance/example_submission/results/zpu1x/ncf/result_62.txt
deleted file mode 100644
index dde92ddc1..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_62.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-82
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_63.txt b/compliance/example_submission/results/zpu1x/ncf/result_63.txt
deleted file mode 100644
index 76a8b2b70..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_63.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-83
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_64.txt b/compliance/example_submission/results/zpu1x/ncf/result_64.txt
deleted file mode 100644
index 871727de1..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_64.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-84
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_65.txt b/compliance/example_submission/results/zpu1x/ncf/result_65.txt
deleted file mode 100644
index a862eb844..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_65.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-85
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_66.txt b/compliance/example_submission/results/zpu1x/ncf/result_66.txt
deleted file mode 100644
index 8cf5c1a22..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_66.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-86
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_67.txt b/compliance/example_submission/results/zpu1x/ncf/result_67.txt
deleted file mode 100644
index 84df3526d..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_67.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-87
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_68.txt b/compliance/example_submission/results/zpu1x/ncf/result_68.txt
deleted file mode 100644
index d22307c42..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_68.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-88
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_69.txt b/compliance/example_submission/results/zpu1x/ncf/result_69.txt
deleted file mode 100644
index 8643cf6de..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_69.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-89
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_7.txt b/compliance/example_submission/results/zpu1x/ncf/result_7.txt
deleted file mode 100644
index f64f5d8d8..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_7.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-27
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_70.txt b/compliance/example_submission/results/zpu1x/ncf/result_70.txt
deleted file mode 100644
index d61f00d8c..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_70.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-90
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_71.txt b/compliance/example_submission/results/zpu1x/ncf/result_71.txt
deleted file mode 100644
index 7fe4e495f..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_71.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-91
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_72.txt b/compliance/example_submission/results/zpu1x/ncf/result_72.txt
deleted file mode 100644
index cd5b02527..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_72.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-92
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_73.txt b/compliance/example_submission/results/zpu1x/ncf/result_73.txt
deleted file mode 100644
index c67f579c9..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_73.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-93
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_74.txt b/compliance/example_submission/results/zpu1x/ncf/result_74.txt
deleted file mode 100644
index 49541f721..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_74.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-94
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_75.txt b/compliance/example_submission/results/zpu1x/ncf/result_75.txt
deleted file mode 100644
index 5595fa46c..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_75.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-95
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_76.txt b/compliance/example_submission/results/zpu1x/ncf/result_76.txt
deleted file mode 100644
index f906e1845..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_76.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-96
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_77.txt b/compliance/example_submission/results/zpu1x/ncf/result_77.txt
deleted file mode 100644
index c17e934b5..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_77.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-97
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_78.txt b/compliance/example_submission/results/zpu1x/ncf/result_78.txt
deleted file mode 100644
index 6529ff889..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_78.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-98
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_79.txt b/compliance/example_submission/results/zpu1x/ncf/result_79.txt
deleted file mode 100644
index 3ad5abd03..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_79.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-99
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_8.txt b/compliance/example_submission/results/zpu1x/ncf/result_8.txt
deleted file mode 100644
index 9902f1784..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_8.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-28
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_80.txt b/compliance/example_submission/results/zpu1x/ncf/result_80.txt
deleted file mode 100644
index 29d6383b5..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_80.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-100
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_81.txt b/compliance/example_submission/results/zpu1x/ncf/result_81.txt
deleted file mode 100644
index 398050c62..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_81.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-101
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_82.txt b/compliance/example_submission/results/zpu1x/ncf/result_82.txt
deleted file mode 100644
index 257e56326..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_82.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-102
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_83.txt b/compliance/example_submission/results/zpu1x/ncf/result_83.txt
deleted file mode 100644
index a9c8fe829..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_83.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-103
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_84.txt b/compliance/example_submission/results/zpu1x/ncf/result_84.txt
deleted file mode 100644
index b16e5f75e..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_84.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-104
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_85.txt b/compliance/example_submission/results/zpu1x/ncf/result_85.txt
deleted file mode 100644
index f96ac0672..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_85.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-105
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_86.txt b/compliance/example_submission/results/zpu1x/ncf/result_86.txt
deleted file mode 100644
index fe4afb0df..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_86.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-106
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_87.txt b/compliance/example_submission/results/zpu1x/ncf/result_87.txt
deleted file mode 100644
index e34885bbc..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_87.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-107
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_88.txt b/compliance/example_submission/results/zpu1x/ncf/result_88.txt
deleted file mode 100644
index 3b20426c0..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_88.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-108
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_89.txt b/compliance/example_submission/results/zpu1x/ncf/result_89.txt
deleted file mode 100644
index e2a9fee00..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_89.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-109
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_9.txt b/compliance/example_submission/results/zpu1x/ncf/result_9.txt
deleted file mode 100644
index f04c001f3..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_9.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-29
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_90.txt b/compliance/example_submission/results/zpu1x/ncf/result_90.txt
deleted file mode 100644
index bc6298e80..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_90.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-110
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_91.txt b/compliance/example_submission/results/zpu1x/ncf/result_91.txt
deleted file mode 100644
index 58c9bdf9d..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_91.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-111
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_92.txt b/compliance/example_submission/results/zpu1x/ncf/result_92.txt
deleted file mode 100644
index 194b81caa..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_92.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-112
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_93.txt b/compliance/example_submission/results/zpu1x/ncf/result_93.txt
deleted file mode 100644
index a9c8fe829..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_93.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-103
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_94.txt b/compliance/example_submission/results/zpu1x/ncf/result_94.txt
deleted file mode 100644
index dee79f109..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_94.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-114
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_95.txt b/compliance/example_submission/results/zpu1x/ncf/result_95.txt
deleted file mode 100644
index ee977b5ec..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_95.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-115
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_96.txt b/compliance/example_submission/results/zpu1x/ncf/result_96.txt
deleted file mode 100644
index 4699eb3cc..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_96.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-116
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_97.txt b/compliance/example_submission/results/zpu1x/ncf/result_97.txt
deleted file mode 100644
index 5bc6609e3..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_97.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-117
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_98.txt b/compliance/example_submission/results/zpu1x/ncf/result_98.txt
deleted file mode 100644
index 415196e47..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_98.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-118
diff --git a/compliance/example_submission/results/zpu1x/ncf/result_99.txt b/compliance/example_submission/results/zpu1x/ncf/result_99.txt
deleted file mode 100644
index 078fa0fe5..000000000
--- a/compliance/example_submission/results/zpu1x/ncf/result_99.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-119
diff --git a/compliance/example_submission/results/zpu1x/resnet/result_0.txt b/compliance/example_submission/results/zpu1x/resnet/result_0.txt
deleted file mode 100644
index f599e28b8..000000000
--- a/compliance/example_submission/results/zpu1x/resnet/result_0.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-10
diff --git a/compliance/example_submission/results/zpu1x/resnet/result_1.txt b/compliance/example_submission/results/zpu1x/resnet/result_1.txt
deleted file mode 100644
index b4de39476..000000000
--- a/compliance/example_submission/results/zpu1x/resnet/result_1.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-11
diff --git a/compliance/example_submission/results/zpu1x/resnet/result_2.txt b/compliance/example_submission/results/zpu1x/resnet/result_2.txt
deleted file mode 100644
index 48082f72f..000000000
--- a/compliance/example_submission/results/zpu1x/resnet/result_2.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-12
diff --git a/compliance/example_submission/results/zpu1x/resnet/result_3.txt b/compliance/example_submission/results/zpu1x/resnet/result_3.txt
deleted file mode 100644
index b1bd38b62..000000000
--- a/compliance/example_submission/results/zpu1x/resnet/result_3.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-13
diff --git a/compliance/example_submission/results/zpu1x/resnet/result_4.txt b/compliance/example_submission/results/zpu1x/resnet/result_4.txt
deleted file mode 100644
index 8351c1939..000000000
--- a/compliance/example_submission/results/zpu1x/resnet/result_4.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-14
diff --git a/compliance/example_submission/results/zpu8x/entry.json b/compliance/example_submission/results/zpu8x/entry.json
deleted file mode 100644
index 00cc63d72..000000000
--- a/compliance/example_submission/results/zpu8x/entry.json
+++ /dev/null
@@ -1,29 +0,0 @@
-{
-    "division": "open",    
-    "status": "on-premise",    
-    "hardware": "1 ZPU",    
-    "framework": "TensorFlow 1.12",    
-    "power": "https://mlperf.org/",    
-    "notes": "ZPUs are the best",    
-    "interconnect": "InfiniBand ring",    
-    "os": "Ubuntu 18.10",    
-    "libraries": "cuDNN 7.3",    
-    "compilers": "N/A",    
-    "nodes": [
-            {
-                "num_nodes": 1,
-                "cpu": "Skylake X",
-                "num_cores": 18,
-                "num_vcpus": 0,
-                "accelerator": "zpu",
-                "num_accelerators": 8,
-                "sys_mem_size": 512,
-                "sys_storage_type": "SSD",
-                "sys_storage_size": 1024,
-                "cpu_accel_interconnect": "zpu-intercon",
-                "network_card": "100 Gb InfiniBand",
-                "num_network_cards": 1,
-                "notes": "this is not a real product" 
-            }
-    ]
-}
diff --git a/compliance/example_submission/results/zpu8x/resnet/result_0.txt b/compliance/example_submission/results/zpu8x/resnet/result_0.txt
deleted file mode 100644
index 7ed6ff82d..000000000
--- a/compliance/example_submission/results/zpu8x/resnet/result_0.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-5
diff --git a/compliance/example_submission/results/zpu8x/resnet/result_1.txt b/compliance/example_submission/results/zpu8x/resnet/result_1.txt
deleted file mode 100644
index 1e8b31496..000000000
--- a/compliance/example_submission/results/zpu8x/resnet/result_1.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-6
diff --git a/compliance/example_submission/results/zpu8x/resnet/result_2.txt b/compliance/example_submission/results/zpu8x/resnet/result_2.txt
deleted file mode 100644
index 7f8f011eb..000000000
--- a/compliance/example_submission/results/zpu8x/resnet/result_2.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-7
diff --git a/compliance/example_submission/results/zpu8x/resnet/result_3.txt b/compliance/example_submission/results/zpu8x/resnet/result_3.txt
deleted file mode 100644
index 45a4fb75d..000000000
--- a/compliance/example_submission/results/zpu8x/resnet/result_3.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-8
diff --git a/compliance/example_submission/results/zpu8x/resnet/result_4.txt b/compliance/example_submission/results/zpu8x/resnet/result_4.txt
deleted file mode 100644
index ec635144f..000000000
--- a/compliance/example_submission/results/zpu8x/resnet/result_4.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-9
diff --git a/compliance/example_submission/submission.json b/compliance/example_submission/submission.json
deleted file mode 100644
index 8080787eb..000000000
--- a/compliance/example_submission/submission.json
+++ /dev/null
@@ -1,4 +0,0 @@
-{
-    "org": "Test Inc.",
-    "poc_email": "johndoe@test.com"
-}
diff --git a/compliance/mllog/README.md b/compliance/mllog/README.md
deleted file mode 100644
index b778414d4..000000000
--- a/compliance/mllog/README.md
+++ /dev/null
@@ -1,31 +0,0 @@
-# MLLog
-
-## Install
-
-Use one of the following ways to install.
-
-- For development, you may download the latest version and install from local path:
-
-```sh
-git clone https://github.com/mlperf/training.git
-pip install -e training/compliance
-```
-
-- Install from github at a specific commit (Replace COMMIT_HASH with actual commit hash, the double quotes are needed):
-  ```
-  pip install "git+https://github.com/mlperf/training@COMMIT_HASH#subdirectory=compliance"
-  ```
-
-Uninstall:
-
-```sh
-pip uninstall mlperf_compliance
-```
-
-## Use
-
-[Here](examples/dummy_example.py) is an example of how to use the `mllog` package in benchmark codes. You can run the example by:
-
-```sh
-python3 -m mllog.examples.dummy_example 
-```
diff --git a/compliance/mllog/__init__.py b/compliance/mllog/__init__.py
deleted file mode 100644
index abcc2d62c..000000000
--- a/compliance/mllog/__init__.py
+++ /dev/null
@@ -1,97 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-import logging
-import sys
-import threading
-
-from mllog import mllog
-
-
-# _lock is for serializing access to shared data structures in this module.
-_lock = threading.RLock()
-
-# mllogger is a logger shareable across modules.
-mllogger = mllog.MLLogger()
-
-
-def get_mllogger():
-  """Get the shared logger."""
-  return mllogger
-
-def config(**kwargs):
-  """Configure the shared logger.
-  Optional keyword arguments:
-    logger: a logging.Logger instance. Customize the logger to change
-      the logging behavior (e.g. logging to a file, etc.)
-    filename: a log file to use. If set, a default file handler will be added
-      to the logger so it can log to the specified file. For more advanced
-      customizations, please set the 'logger' parameter instead.
-    default_namespace: the default namespace to use if one isn't provided.
-    default_stack_offset: the default depth to go into the stack to find the
-      call site.
-    default_clear_line: the default behavior of line clearing (i.e. print
-      an extra new line to clear any pre-existing text in the log line).
-    root_dir: directory prefix which will be trimmed when reporting calling
-      file for logging.
-  """
-  if _lock:
-    _lock.acquire()
-  try:
-    logger = kwargs.pop("logger", None)
-    if logger is not None:
-      if not isinstance(logger, logging.Logger):
-        raise ValueError("'logger' must be an instance of 'logging.Logger'.")
-      if logger.name == mllogger.logger.name:
-        raise ValueError("'logger' should not be the same as the default " +
-                         "logger to avoid unexpected behavior. Consider " +
-                         "using a different name for the logger.")
-      mllogger.logger = logger
-
-    log_file = kwargs.pop("filename", None)
-    if log_file is not None:
-      if not isinstance(log_file, str):
-        raise ValueError("'filename' must be a string.")
-      _file_handler = logging.FileHandler(log_file)
-      _file_handler.setLevel(logging.INFO)
-      mllogger.logger.addHandler(_file_handler)
-
-    default_namespace = kwargs.pop("default_namespace", None)
-    if default_namespace is not None:
-      if not isinstance(default_namespace, str):
-        raise ValueError("'default_namespace' must be a string.")
-      mllogger.default_namespace = default_namespace
-
-    default_stack_offset = kwargs.pop("default_stack_offset", None)
-    if default_stack_offset is not None:
-      if not isinstance(default_stack_offset, int):
-        raise ValueError("'default_stack_offset' must be an integer.")
-      mllogger.default_stack_offset = default_stack_offset
-
-    default_clear_line = kwargs.pop("default_clear_line", None)
-    if default_clear_line is not None:
-      if not isinstance(default_clear_line, bool):
-        raise ValueError("'default_clear_line' must be a boolean value.")
-      mllogger.default_clear_line = default_clear_line
-
-    root_dir = kwargs.pop("root_dir", None)
-    if root_dir is not None:
-      if not isinstance(root_dir, str):
-        raise ValueError("'root_dir' must be a string.")
-      mllogger.root_dir = root_dir
-
-  finally:
-    if _lock:
-      _lock.release()
diff --git a/compliance/mllog/constants.py b/compliance/mllog/constants.py
deleted file mode 100644
index 2c10e51f3..000000000
--- a/compliance/mllog/constants.py
+++ /dev/null
@@ -1,108 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Master list of constants in MLPerf log
-"""
-
-# NOTE: Keep string values in alphabetical order under each section.
-
-# Constant values - log settings
-DEFAULT_LOGGER_NAME = "mllog_default"
-DEFAULT_NAMESPACE = ""
-
-# Constant values - log event type
-INTERVAL_END = "INTERVAL_END"
-INTERVAL_START = "INTERVAL_START"
-POINT_IN_TIME = "POINT_IN_TIME"
-
-# Constant values - benchmark name
-GNMT = "gnmt"
-MASKRCNN = "maskrcnn"
-MINIGO = "minigo"
-NCF = "ncf"
-RESNET = "resnet"
-SSD = "ssd"
-TRANSFORMER = "transformer"
-
-# Constant values - model info
-ADAM = "adam"
-LARS = "lars"
-LAZY_ADAM = "lazy_adam"
-SGD = "sgd"
-
-# Constant values - metadata info
-ABORTED = "aborted"
-SUCCESS = "success"
-
-# Log keys - submission info
-SUBMISSION_BENCHMARK = "submission_benchmark"
-SUBMISSION_DIVISION = "submission_division"
-SUBMISSION_ENTRY = "submission_entry"
-SUBMISSION_ORG = "submission_org"
-SUBMISSION_PLATFORM = "submission_platform"
-SUBMISSION_POC_NAME = "submission_poc_name"
-SUBMISSION_POC_EMAIL = "submission_poc_email"
-SUBMISSION_STATUS = "submission_status"
-
-# Log keys - timing info
-BLOCK_START = "block_start"
-BLOCK_STOP = "block_stop"
-EPOCH_START = "epoch_start"
-EPOCH_STOP = "epoch_stop"
-EVAL_START = "eval_start"
-EVAL_STOP = "eval_stop"
-INIT_START = "init_start"
-INIT_STOP = "init_stop"
-RUN_START = "run_start"
-RUN_STOP = "run_stop"
-
-# Log keys - common run info
-CACHE_CLEAR = "cache_clear"
-EVAL_ACCURACY = "eval_accuracy"
-
-# Log kyes - model hyperparameters
-GLOBAL_BATCH_SIZE = "global_batch_size"
-LARS_EPSILON = "lars_epsilon"
-LARS_OPT_END_LR = "lars_opt_end_learning_rate"
-LARS_OPT_LR_DECAY_POLY_POWER = "lars_opt_learning_rate_decay_poly_power"
-LARS_OPT_LR_DECAY_STEPS = "lars_opt_learning_rate_decay_steps"
-LARS_OPT_WEIGHT_DECAY = "lars_opt_weight_decay"
-MAX_SAMPLES = "max_samples"
-MAX_SEQUENCE_LENGTH = "max_sequence_length"
-MODEL_BN_SPAN = "model_bn_span"
-NUM_IMAGE_CANDIDATES = "num_image_candidates"
-OPT_ADAM_BETA_1 = "opt_adam_beta_1"
-OPT_ADAM_BETA_2 = "opt_adam_beta_2"
-OPT_ADAM_EPSILON = "opt_adam_epsilon"
-OPT_NAME = "opt_name"
-OPT_BASE_LR = "opt_base_learning_rate"
-OPT_LR_ALT_DECAY_FUNC = "opt_learning_rate_alt_decay_func"
-OPT_LR_ALT_WARMUP_FUNC = "opt_learning_rate_alt_warmup_func"
-OPT_LR_DECAY_BOUNDARY_EPOCHS = "opt_learning_rate_decay_boundary_epochs"
-OPT_LR_DECAY_BOUNDARY_STEPS = "opt_learning_rate_decay_boundary_steps"
-OPT_LR_DECAY_FACTOR = "opt_learning_rate_decay_factor"
-OPT_LR_DECAY_INTERVAL = "opt_learning_rate_decay_interval"
-OPT_LR_DECAY_STEPS = "opt_learning_rate_decay_steps"
-OPT_LR_REMAIN_STEPS = "opt_learning_rate_remain_steps"
-OPT_LR_WARMUP_EPOCHS = "opt_learning_rate_warmup_epochs"
-OPT_LR_WARMUP_FACTOR = "opt_learning_rate_warmup_factor"
-OPT_LR_WARMUP_STEPS = "opt_learning_rate_warmup_steps"
-OPT_WEIGHT_DECAY = "opt_weight_decay"
-
-# Log metadata keys
-EPOCH_COUNT = "epoch_count"
-EPOCH_NUM = "epoch_num"
-FIRST_EPOCH_NUM = "first_epoch_num"
-STATUS = "status"
diff --git a/compliance/mllog/examples/__init__.py b/compliance/mllog/examples/__init__.py
deleted file mode 100644
index 658ad1a17..000000000
--- a/compliance/mllog/examples/__init__.py
+++ /dev/null
@@ -1,14 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
diff --git a/compliance/mllog/examples/dummy_example.py b/compliance/mllog/examples/dummy_example.py
deleted file mode 100644
index 61b6171fa..000000000
--- a/compliance/mllog/examples/dummy_example.py
+++ /dev/null
@@ -1,87 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-import logging
-import os
-import sys
-
-import mllog
-from mllog import constants
-
-
-def dummy_example():
-  """Example usage of mllog"""
-
-  # Get the mllogger instance, this needs to be called in every module that
-  # needs logging
-  mllogger = mllog.get_mllogger()
-
-  # Customize mllogger configuration
-  # These configurations only need to be set Once in your entire program.
-  # Try tweaking the following configurations to see the difference.
-  #   logger: Customize the underlying logger to change the logging behavior.
-  #   filename: a log file to use. If set, a default file handler will be added
-  #     to the logger so it can log to the specified file. For more advanced
-  #     customizations, please set the 'logger' parameter instead.
-  #   default_namespace: the default namespace to use if one isn't provided.
-  #   default_stack_offset: the default depth to go into the stack to find
-  #     the call site.
-  #   default_clear_line: the default behavior of line clearing (i.e. print
-  #     an extra new line to clear any pre-existing text in the log line).
-  #   root_dir: directory prefix which will be trimmed when reporting calling
-  #     file for logging.
-
-  # Customize the underlying logger to use a file in addition to stdout.
-  # 1. Simple way
-  # Provide a filename, this adds a log file with default behavior.
-  mllog.config(filename="example_simple.log")
-  # 2. Advanced way
-  # You may pass a logging.Logger instance to mllog.config().
-  # To use the advanced way, comment out the "Simple way" above and uncomment
-  # the followings:
-  #
-  # # Notice that proper log level needs to be set for both logger and handler.
-  # logger = logging.getLogger("custom_logger")
-  # logger.setLevel(logging.DEBUG)
-  # # add file handler for file logging
-  # _file_handler = logging.FileHandler("example_advanced.log")
-  # _file_handler.setLevel(logging.DEBUG)
-  # logger.addHandler(_file_handler)
-  # # add stream handler for stdout logging
-  # _stream_handler = logging.StreamHandler(stream=sys.stdout)
-  # _stream_handler.setLevel(logging.INFO)
-  # logger.addHandler(_stream_handler)
-  # mllog.config(logger=logger)
-
-  # Set other logger configurations
-  mllog.config(
-      default_namespace = "worker1",
-      default_stack_offset = 1,
-      default_clear_line = False,
-      root_dir = os.path.normpath(
-        os.path.join(os.path.dirname(os.path.realpath(__file__)), "..", "..")))
-
-  # Example log messages
-  # The methods to use are "start", "end", and "event".
-  # You may check out the detailed APIs in mllog.mllog.
-  # Try to use the keys from mllog.constants to avoid wrong keys.
-  mllogger.start(key=constants.RUN_START)
-  mllogger.event(key=constants.GLOBAL_BATCH_SIZE, value=1024)
-  mllogger.event(key=constants.EVAL_ACCURACY, value=0.99, clear_line=True)
-  mllogger.end(key=constants.RUN_STOP)
-
-
-if __name__ == "__main__":
-  dummy_example()
diff --git a/compliance/mllog/mllog.py b/compliance/mllog/mllog.py
deleted file mode 100644
index 506a308ad..000000000
--- a/compliance/mllog/mllog.py
+++ /dev/null
@@ -1,280 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Convenience function for logging compliance tags to stdout."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import collections
-import datetime
-import inspect
-import json
-import logging
-import os
-import re
-import sys
-import time
-import timeit
-
-from mllog import constants
-
-LOG_TEMPLATE = ':::MLLOG {log_json}'
-
-
-def get_caller(stack_index=2, root_dir=None):
-  """Get caller's file and line number information.
-  Args:
-    stack_index: a stack_index of 2 will provide the caller of the
-      function calling this function. Notice that stack_index of 2
-      or more will fail if called from global scope.
-    root_dir: the root dir prefixed to the file name. The root_dir
-      will be trimmed from the file name in the output.
-  Returns:
-    Call site info in a dictionary with these fields:
-      "file": (string) file path
-      "lineno": (int) line number
-  """
-  caller = inspect.getframeinfo(inspect.stack()[stack_index][0])
-
-  # Trim the filenames for readability.
-  filename = caller.filename
-  if root_dir is not None:
-    filename = re.sub('^' + root_dir + '/', '', filename)
-  return {"file": filename, "lineno": caller.lineno}
-
-
-def _now_as_str():
-  """Returns the current time as a human readable string."""
-  return datetime.datetime.now().strftime('%H:%M:%S.%f')
-
-
-def _current_time_ms():
-  """Returns current milliseconds since epoch."""
-  return int(time.time() * 1e3)
-
-
-def _try_float(value):
-  """Tries to convert a value to a float (otherwise returns the input)."""
-  try:
-    return float(value)
-  except Exception:  # pylint:disable=broad-except
-    return value
-
-
-def _to_ordered_json(kv_pairs):
-  """Convert a list of (key, value) pairs to a serialized json message.
-  Args:
-    kv_pairs: List of (key, value) tuples, ordered by how they should be
-      presented.
-  Returns:
-    string - The serialized json string with fields in the same order as
-      kv_pairs. If there were any key/value pairs that couldn't be properly
-      converted to json, returns an error string.
-  """
-  d = collections.OrderedDict()
-  # json.dumps() works differently in python 2 and 3 for boolean keys.
-  # py2: json.dumps({True: 'a', False: 'b'}) -> '{"True": "a", "False": "b"}'
-  # py3: json.dumps({True: 'a', False: 'b'}) -> '{"true": "a", "false": "b"}'
-  # We standardize to match python 3 output.
-  # Note that we don't need to account for this in boolean values.
-  # py2 and 3: json.dumps({'a': True, 'b': False}) -> '{"a": true, "b": false}'
-  for key, value in kv_pairs:
-    if isinstance(key, bool):
-      if key:
-        d['true'] = value
-      else:
-        d['false'] = value
-    elif key == 'value':
-      # See if we can convert the 'value' field to a float where possible.
-      # This is to mostly handle np.float values which cannot be json encoded.
-      d[key] = _try_float(value)
-    else:
-      d[key] = value
-  try:
-    return json.dumps(d)
-  except Exception:  # pylint:disable=broad-except
-    return '[convert-error: {kv_str}]'.format(kv_str=str(kv_pairs))
-
-
-def _encode_log(namespace, time_ms, event_type, key, value, metadata):
-  """Encodes an MLEvent as a string log line.
-  Args:
-    namespace: provides structure, e.g. "GPU0".
-    time_ms: milliseconds since unix epoch.
-    event_type: one of: 'INTERVAL_START', 'INTERVAL_END', 'POINT_IN_TIME'
-    key: the name of the thing being logged.
-    value: a json value.
-    metadata: a json value.
-  Returns:
-    A string log like, i.e. ":::MLLog { ..."
-  """
-  # preserve the order of key-values
-  ordered_key_val_pairs = [
-    ('namespace', namespace),
-    ('time_ms', time_ms),
-    ('event_type', event_type),
-    ('key', key),
-    ('value', value),
-    ('metadata', metadata)
-  ]
-  encoded = _to_ordered_json(ordered_key_val_pairs)
-  return LOG_TEMPLATE.format(log_json=encoded)
-
-
-class MLLogger(object):
-  """MLPerf logging helper."""
-
-  def __init__(self,
-               logger=None,
-               default_namespace=constants.DEFAULT_NAMESPACE,
-               default_stack_offset=1,
-               default_clear_line=False,
-               root_dir=None):
-    """Create a new MLLogger.
-    Args:
-      logger: a logging.Logger instance. If not specified, a default logger
-        will be used which prints to stdout. Customize the logger to change
-        the logging behavior (e.g. logging to a file, etc.)
-      default_namespace: the default namespace to use if one isn't provided.
-      default_stack_offset: the default depth to go into the stack to find the
-        call site. Default value is 1.
-      default_clear_line: the default behavior of line clearing (i.e. print
-        an extra new line to clear any pre-existing text in the log line).
-      root_dir: directory prefix which will be trimmed when reporting calling
-        file for logging.
-    """
-    if logger is None:
-      self.logger = self._get_default_logger()
-    elif not isinstance(logger, logging.Logger):
-      raise ValueError("logger must be a `logging.Logger` instance.")
-    else:
-      self.logger = logger
-
-    self.default_namespace = default_namespace
-    self.default_stack_offset = default_stack_offset
-    self.default_clear_line = default_clear_line
-    self.root_dir = root_dir
-
-  def _get_default_logger(self):
-    """Create a default logger.
-    The default logger prints INFO level messages to stdout.
-    """
-    logger = logging.getLogger(constants.DEFAULT_LOGGER_NAME)
-    logger.setLevel(logging.INFO)
-    _stream_handler = logging.StreamHandler(stream=sys.stdout)
-    _stream_handler.setLevel(logging.INFO)
-    logger.addHandler(_stream_handler)
-    return logger
-
-  def _do_log(self, level, message, clear_line=False):
-    if clear_line:
-      message = '\n' + message
-    self.logger.log(level, message)
-
-  def _log_helper(self, event_type, key, value=None, metadata=None,
-        namespace=None, time_ms=None, stack_offset=None, clear_line=None):
-    """Log an event."""
-    if namespace is None:
-      namespace = self.default_namespace
-    if time_ms is None:
-      time_ms = _current_time_ms()
-    if stack_offset is None:
-      stack_offset = self.default_stack_offset
-    if clear_line is None:
-      clear_line = self.default_clear_line
-
-    log_metadata = {}
-    log_metadata.update(get_caller(2 + stack_offset, root_dir=self.root_dir))
-    if metadata:
-      if not isinstance(metadata, dict):
-        self._do_log(logging.WARNING, "Metadata is not dictionary, ignored.",
-                     clear_line=True)
-      else:
-        overlap_keys = set(log_metadata.keys()).intersection(metadata.keys())
-        if overlap_keys:
-          self._do_log(logging.WARNING,
-              "Metadata fields overridden: {}".format(", ".join(overlap_keys)),
-              clear_line=True)
-        log_metadata.update(metadata)
-
-    log_line = _encode_log(
-        namespace,
-        time_ms,
-        event_type,
-        key,
-        value,
-        log_metadata)
-
-    self._do_log(logging.INFO, log_line, clear_line)
-
-  def start(self, key, value=None, metadata=None, namespace=None, time_ms=None,
-            stack_offset=None, clear_line=None):
-    """Start an time interval in the log.
-    All intervals which are started must be ended. This interval must be
-    ended before a new interval with the same key and namespace can be started.
-    Args:
-      key: the key for the event, e.g. "mlperf.training"
-      value: the value to log at the start of the interval.
-      metadata: a dictionary containing metadata corresponding to the log event.
-      namespace: override the default namespace.
-      time_ms: the time in milliseconds, or None for current time.
-      stack_offset: override the default stack offset, i.e. the depth to go
-        into the stack to find the call site.
-      clear_line: override the default line clearing behavior, i.e. whether to
-        print an extra new line to clear pre-existing text in the log line.
-    """
-    self._log_helper(constants.INTERVAL_START, key, value,
-                     metadata=metadata, namespace=namespace, time_ms=time_ms,
-                     stack_offset=stack_offset, clear_line=clear_line)
-
-  def end(self, key, value=None, metadata=None, namespace=None, time_ms=None,
-          stack_offset=None, clear_line=None):
-    """End a time interval in the log.
-    Ends an interval which was already started with the same key and in the
-    same namespace.
-    Args:
-      key: the same log key which was passed to start().
-      value: the value to log at the end of the interval.
-      metadata: a dictionary containing metadata corresponding to the log event.
-      namespace: optional override of the default namespace.
-      time_ms: the time in milliseconds, or None for current time.
-      stack_offset: override the default stack offset, i.e. the depth to go
-        into the stack to find the call site.
-      clear_line: override the default line clearing behavior, i.e. whether to
-        print an extra new line to clear pre-existing text in the log line.
-    """
-    self._log_helper(constants.INTERVAL_END, key, value,
-                     metadata=metadata, namespace=namespace, time_ms=time_ms,
-                     stack_offset=stack_offset, clear_line=clear_line)
-
-  def event(self, key, value=None, metadata=None, namespace=None, time_ms=None,
-            stack_offset=None, clear_line=None):
-    """Log a point in time event.
-    The event does not have an associated duration like an interval has.
-    Args:
-      key: the "name" of the event.
-      value: the event data itself.
-      metadata: a dictionary containing metadata corresponding to the log event.
-      namespace: optional override of the default namespace.
-      time_ms: the time in milliseconds, or None for current time.
-      stack_offset: override the default stack offset, i.e. the depth to go
-        into the stack to find the call site.
-      clear_line: override the default line clearing behavior, i.e. whether to
-        print an extra new line to clear pre-existing text in the log line.
-    """
-    self._log_helper(constants.POINT_IN_TIME, key, value,
-                     metadata=metadata, namespace=namespace, time_ms=time_ms,
-                     stack_offset=stack_offset, clear_line=clear_line)
diff --git a/compliance/mlperf_compliance/__init__.py b/compliance/mlperf_compliance/__init__.py
deleted file mode 100644
index abc80bb29..000000000
--- a/compliance/mlperf_compliance/__init__.py
+++ /dev/null
@@ -1,1 +0,0 @@
-from . import mlperf_log
diff --git a/compliance/mlperf_compliance/_gnmt_tags.py b/compliance/mlperf_compliance/_gnmt_tags.py
deleted file mode 100644
index 097e97b83..000000000
--- a/compliance/mlperf_compliance/_gnmt_tags.py
+++ /dev/null
@@ -1,51 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Keys which only appear in GNMT RNN Translation.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-# Loss smoothing factor
-MODEL_HP_LOSS_SMOOTHING = "model_hp_loss_smoothing"
-
-# Number of layers in encoder and in decoder
-MODEL_HP_NUM_LAYERS = "model_hp_num_layers"
-
-# RNN hidden size
-MODEL_HP_HIDDEN_SIZE = "model_hp_hidden_size"
-
-# Dropout
-MODEL_HP_DROPOUT = "model_hp_dropout"
-
-# Beam size for beam search
-EVAL_HP_BEAM_SIZE = "eval_hp_beam_size"
-
-# Maximum sequence length for training
-TRAIN_HP_MAX_SEQ_LEN = "train_hp_max_sequence_length"
-
-# Maximum sequence length for evaluation
-EVAL_HP_MAX_SEQ_LEN = "eval_hp_max_sequence_length"
-
-# Length normalization constant for beam search
-EVAL_HP_LEN_NORM_CONST = "eval_hp_length_normalization_constant"
-
-# Length normalization factor for beam search
-EVAL_HP_LEN_NORM_FACTOR = "eval_hp_length_normalization_factor"
-
-# Coverage penalty factor for beam search
-EVAL_HP_COV_PENALTY_FACTOR = "eval_hp_coverage_penalty_factor"
diff --git a/compliance/mlperf_compliance/_maskrcnn_tags.py b/compliance/mlperf_compliance/_maskrcnn_tags.py
deleted file mode 100644
index 1ec5ecdf1..000000000
--- a/compliance/mlperf_compliance/_maskrcnn_tags.py
+++ /dev/null
@@ -1,53 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Keys which only appear in MASKRCNN.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-# Anchor overlap threshop
-FG_IOU_THRESHOLD = "foreground_iou_threshold"
-BG_IOU_THRESHOLD = "background_iou_threshold"
-
-# Top ROIs to be selected before and after NMS
-RPN_PRE_NMS_TOP_N_TRAIN = "rpn_pre_nms_top_n_train"
-RPN_PRE_NMS_TOP_N_TEST = "rpn_pre_nms_top_n_test"
-RPN_POST_NMS_TOP_N_TRAIN = "rpn_post_nms_top_n_train"
-RPN_POST_NMS_TOP_N_TEST = "rpn_post_nms_top_n_test"
-
-#Global batch size during training
-GLOBAL_BATCH_SIZE = "global_batch_size"
-
-# Batch size during eval
-BATCH_SIZE_TEST = "batch_size_test"
-
-
-# Pretrained classifer model
-BACKBONE = "backbone"
-
-# Anchor aspect ratio
-ASPECT_RATIOS = "aspect_ratios"
-
-# Overlap threshold for NMS
-NMS_THRESHOLD = "nms_threshold"
-
-# data pipeline
-MIN_IMAGE_SIZE = "min_image_size"
-MAX_IMAGE_SIZE = "max_image_size"
-RANDOM_FLIP_PROBABILITY = "random_flip_probability"
-INPUT_NORMALIZATION_STD = "input_normalization_std"
diff --git a/compliance/mlperf_compliance/_ncf_tags.py b/compliance/mlperf_compliance/_ncf_tags.py
deleted file mode 100644
index a1235b818..000000000
--- a/compliance/mlperf_compliance/_ncf_tags.py
+++ /dev/null
@@ -1,60 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Keys which only appear in NCF Recommendation.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-# The minimum number of ratings for a user to be included.
-PREPROC_HP_MIN_RATINGS = "preproc_hp_min_ratings"
-
-# The number of false negatives to use during evaluation.
-PREPROC_HP_NUM_EVAL = "preproc_hp_num_eval"
-
-# Are evaluation negatives sampled with replacement?
-PREPROC_HP_SAMPLE_EVAL_REPLACEMENT = "preproc_hp_sample_eval_replacement"
-
-
-# The number of false negatives per postive generated during training.
-INPUT_HP_NUM_NEG = "input_hp_num_neg"
-
-# Are training negatives sampled with replacement?
-INPUT_HP_SAMPLE_TRAIN_REPLACEMENT = "input_hp_sample_train_replacement"
-
-# This tag should be emitted each time the submission begins construction of the
-# false negatives for a trainging epoch.
-INPUT_STEP_TRAIN_NEG_GEN = "input_step_train_neg_gen"
-
-# This tag should be emitted when the evaluation negatives are selected. This
-# should occur only once.
-INPUT_STEP_EVAL_NEG_GEN = "input_step_eval_neg_gen"
-
-# The number of users in the evaluation set. This should be the same as the
-# number of users in the training set.
-EVAL_HP_NUM_USERS = "eval_hp_num_users"
-
-# The number of false negatives per positive which actually appear during
-# evaluation. This should match PREPROC_HP_NUM_EVAL.
-EVAL_HP_NUM_NEG = "eval_hp_num_neg"
-
-
-# The dimensionality of the matrix factorization portion of the model.
-MODEL_HP_MF_DIM = "model_hp_mf_dim"
-
-# The sizes of the fully connected layers in the dense section of the model.
-MODEL_HP_MLP_LAYER_SIZES = "model_hp_mlp_layer_sizes"
-
diff --git a/compliance/mlperf_compliance/_resnet_tags.py b/compliance/mlperf_compliance/_resnet_tags.py
deleted file mode 100644
index 7aa9360a3..000000000
--- a/compliance/mlperf_compliance/_resnet_tags.py
+++ /dev/null
@@ -1,47 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Keys which only appear in ResNet.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-BOTTLENECK_BLOCK = "bottleneck_block"
-
-# The ResNet reference specifies that evaluation occurs once every four epochs.
-# This can result in a quantization penalty for batch sizes which converge on
-# certain epochs. For instance a batch size which tends to converge on epoch 81
-# or 82 would be unduly punished by evaluating at epochs 80 and 84. In order to
-# address this, submissions may select an offset between 0 and 3 for the first
-# evaluation. So in the example above, the submitter could select an offset of
-# 1. In that case the first evaluation would occur on epoch 2, with later
-# evaluations correspondingly offset. Because this would trigger an eval on
-# epoch 82, the submission in this example can exit at a natural time.
-EVAL_EPOCH_OFFSET = "eval_offset"
-
-# ==============================================================================
-# == Topology ==================================================================
-# ==============================================================================
-
-MODEL_HP_INITIAL_MAX_POOL = "model_hp_initial_max_pool"
-MODEL_HP_BEGIN_BLOCK = "model_hp_begin_block"
-MODEL_HP_END_BLOCK = "model_hp_end_block"
-MODEL_HP_BLOCK_TYPE = "model_hp_block_type"
-MODEL_HP_PROJECTION_SHORTCUT = "model_hp_projection_shortcut"
-MODEL_HP_SHORTCUT_ADD = "model_hp_shorcut_add"
-
-MODEL_HP_RESNET_TOPOLOGY = "model_hp_resnet_topology"
diff --git a/compliance/mlperf_compliance/_ssd_tags.py b/compliance/mlperf_compliance/_ssd_tags.py
deleted file mode 100644
index f1a87bea7..000000000
--- a/compliance/mlperf_compliance/_ssd_tags.py
+++ /dev/null
@@ -1,42 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Keys which only appear in SSD.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-# Pretrained classifer model
-BACKBONE = "backbone"
-
-FEATURE_SIZES = "feature_sizes"
-STEPS = "steps"
-SCALES = "scales"
-ASPECT_RATIOS = "aspect_ratios"
-NUM_DEFAULTS_PER_CELL = "num_defaults_per_cell"
-LOC_CONF_OUT_CHANNELS = "loc_conf_out_channels"
-NUM_DEFAULTS = "num_default_boxes"
-
-# Overlap threshold for NMS
-NMS_THRESHOLD = "nms_threshold"
-NMS_MAX_DETECTIONS = "nms_max_detections"
-
-# data pipeline
-NUM_CROPPING_ITERATIONS = "num_cropping_iterations"
-RANDOM_FLIP_PROBABILITY = "random_flip_probability"
-DATA_NORMALIZATION_MEAN = "data_normalization_mean"
-DATA_NORMALIZATION_STD = "data_normalization_std"
diff --git a/compliance/mlperf_compliance/_transformer_tags.py b/compliance/mlperf_compliance/_transformer_tags.py
deleted file mode 100644
index 7355828cf..000000000
--- a/compliance/mlperf_compliance/_transformer_tags.py
+++ /dev/null
@@ -1,35 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Keys which only appear in transformer.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-INPUT_MAX_LENGTH = "input_max_length"
-
-MODEL_HP_INITIALIZER_GAIN = "model_hp_initializer_gain"
-MODEL_HP_VOCAB_SIZE = "model_hp_vocab_size"
-MODEL_HP_NUM_HIDDEN_LAYERS = "model_hp_hidden_layers"
-MODEL_HP_EMBEDDING_SHARED_WEIGHTS = "model_hp_embedding_shared_weights"
-MODEL_HP_ATTENTION_DENSE = "model_hp_attention_dense"
-MODEL_HP_ATTENTION_DROPOUT = "model_hp_attention_dropout"
-MODEL_HP_FFN_OUTPUT_DENSE = "model_hp_ffn_output_dense"
-MODEL_HP_FFN_FILTER_DENSE = "model_hp_ffn_filter_dense"
-MODEL_HP_RELU_DROPOUT = "model_hp_relu_dropout"
-MODEL_HP_LAYER_POSTPROCESS_DROPOUT = "model_hp_layer_postprocess_dropout"
-MODEL_HP_NORM = "model_hp_norm"
-MODEL_HP_SEQ_BEAM_SEARCH = "model_hp_sequence_beam_search"
diff --git a/compliance/mlperf_compliance/mlperf_log.py b/compliance/mlperf_compliance/mlperf_log.py
deleted file mode 100644
index afe340ba0..000000000
--- a/compliance/mlperf_compliance/mlperf_log.py
+++ /dev/null
@@ -1,202 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Convenience function for logging compliance tags to stdout.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import inspect
-import logging
-import json
-import os
-import re
-import sys
-import time
-import uuid
-
-from mlperf_compliance.tags import *
-
-ROOT_DIR_GNMT = None
-ROOT_DIR_MASKRCNN = None
-ROOT_DIR_MINIGO = None
-ROOT_DIR_NCF = None
-
-# Set by imagenet_main.py
-ROOT_DIR_RESNET = None
-
-ROOT_DIR_SSD = None
-
-# Set by transformer_main.py and process_data.py
-ROOT_DIR_TRANSFORMER = None
-
-
-PATTERN = re.compile('[a-zA-Z0-9]+')
-
-LOG_FILE = os.getenv("COMPLIANCE_FILE")
-# create logger with 'spam_application'
-LOGGER = logging.getLogger('mlperf_compliance')
-LOGGER.setLevel(logging.DEBUG)
-
-_STREAM_HANDLER = logging.StreamHandler(stream=sys.stdout)
-_STREAM_HANDLER.setLevel(logging.INFO)
-LOGGER.addHandler(_STREAM_HANDLER)
-
-if LOG_FILE:
-  _FILE_HANDLER = logging.FileHandler(LOG_FILE)
-  _FILE_HANDLER.setLevel(logging.DEBUG)
-  LOGGER.addHandler(_FILE_HANDLER)
-else:
-  _STREAM_HANDLER.setLevel(logging.DEBUG)
-
-
-
-def get_caller(stack_index=2, root_dir=None):
-  ''' Returns file.py:lineno of your caller. A stack_index of 2 will provide
-      the caller of the function calling this function. Notice that stack_index
-      of 2 or more will fail if called from global scope. '''
-  caller = inspect.getframeinfo(inspect.stack()[stack_index][0])
-
-  # Trim the filenames for readability.
-  filename = caller.filename
-  if root_dir is not None:
-    filename = re.sub("^" + root_dir + "/", "", filename)
-  return "%s:%d" % (filename, caller.lineno)
-
-
-def _mlperf_print(key, value=None, benchmark=None, stack_offset=0,
-                  tag_set=None, deferred=False, root_dir=None,
-                  extra_print=False, prefix=""):
-  ''' Prints out an MLPerf Log Line.
-
-  key: The MLPerf log key such as 'CLOCK' or 'QUALITY'. See the list of log keys in the spec.
-  value: The value which contains no newlines.
-  benchmark: The short code for the benchmark being run, see the MLPerf log spec.
-  stack_offset: Increase the value to go deeper into the stack to find the callsite. For example, if this
-                is being called by a wraper/helper you may want to set stack_offset=1 to use the callsite
-                of the wraper/helper itself.
-  tag_set: The set of tags in which key must belong.
-  deferred: The value is not presently known. In that case, a unique ID will
-            be assigned as the value of this call and will be returned. The
-            caller can then include said unique ID when the value is known
-            later.
-  root_dir: Directory prefix which will be trimmed when reporting calling file
-            for compliance logging.
-  extra_print: Print a blank line before logging to clear any text in the line.
-  prefix: String with which to prefix the log message. Useful for
-          differentiating raw lines if stitching will be required.
-
-  Example output:
-    :::MLP-1537375353 MINGO[17] (eval.py:42) QUALITY: 43.7
-  '''
-
-  return_value = None
-
-  if (tag_set is None and not PATTERN.match(key)) or key not in tag_set:
-    raise ValueError('Invalid key for MLPerf print: ' + str(key))
-
-  if value is not None and deferred:
-    raise ValueError("deferred is set to True, but a value was provided")
-
-  if deferred:
-    return_value = str(uuid.uuid4())
-    value = "DEFERRED: {}".format(return_value)
-
-  if value is None:
-    tag = key
-  else:
-    str_json = json.dumps(value)
-    tag = '{key}: {value}'.format(key=key, value=str_json)
-
-  callsite = get_caller(2 + stack_offset, root_dir=root_dir)
-  now = time.time()
-
-  message = '{prefix}:::MLPv0.5.0 {benchmark} {secs:.9f} ({callsite}) {tag}'.format(
-      prefix=prefix, secs=now, benchmark=benchmark, callsite=callsite, tag=tag)
-
-  if extra_print:
-    print() # There could be prior text on a line
-
-  if tag in STDOUT_TAG_SET:
-    LOGGER.info(message)
-  else:
-    LOGGER.debug(message)
-
-  return return_value
-
-
-GNMT_TAG_SET = set(GNMT_TAGS)
-def gnmt_print(key, value=None, stack_offset=1, deferred=False, prefix=""):
-  return _mlperf_print(key=key, value=value, benchmark=GNMT,
-                       stack_offset=stack_offset, tag_set=GNMT_TAG_SET,
-                       deferred=deferred, root_dir=ROOT_DIR_GNMT)
-
-
-MASKRCNN_TAG_SET = set(MASKRCNN_TAGS)
-def maskrcnn_print(key, value=None, stack_offset=1, deferred=False,
-    extra_print=True, prefix=""):
-  return _mlperf_print(key=key, value=value, benchmark=MASKRCNN,
-                       stack_offset=stack_offset, tag_set=MASKRCNN_TAG_SET,
-                       deferred=deferred, extra_print=extra_print,
-                       root_dir=ROOT_DIR_MASKRCNN, prefix=prefix)
-
-
-MINIGO_TAG_SET = set(MINIGO_TAGS)
-def minigo_print(key, value=None, stack_offset=1, deferred=False, prefix=""):
-  return _mlperf_print(key=key, value=value, benchmark=MINIGO,
-                       stack_offset=stack_offset, tag_set=MINIGO_TAG_SET,
-                       deferred=deferred, root_dir=ROOT_DIR_MINIGO,
-                       prefix=prefix)
-
-
-NCF_TAG_SET = set(NCF_TAGS)
-def ncf_print(key, value=None, stack_offset=1, deferred=False,
-              extra_print=True, prefix=""):
-  # Extra print is needed for the reference NCF because of tqdm.
-  return _mlperf_print(key=key, value=value, benchmark=NCF,
-                       stack_offset=stack_offset, tag_set=NCF_TAG_SET,
-                       deferred=deferred, extra_print=extra_print,
-                       root_dir=ROOT_DIR_NCF, prefix=prefix)
-
-
-RESNET_TAG_SET = set(RESNET_TAGS)
-def resnet_print(key, value=None, stack_offset=1, deferred=False, prefix=""):
-  return _mlperf_print(key=key, value=value, benchmark=RESNET,
-                       stack_offset=stack_offset, tag_set=RESNET_TAG_SET,
-                       deferred=deferred, root_dir=ROOT_DIR_RESNET,
-                       prefix=prefix)
-
-
-SSD_TAG_SET = set(SSD_TAGS)
-def ssd_print(key, value=None, stack_offset=1, deferred=False,
-              extra_print=True, prefix=""):
-  return _mlperf_print(key=key, value=value, benchmark=SSD,
-                       stack_offset=stack_offset, tag_set=SSD_TAG_SET,
-                       deferred=deferred, extra_print=extra_print,
-                       root_dir=ROOT_DIR_SSD, prefix=prefix)
-
-
-TRANSFORMER_TAG_SET = set(TRANSFORMER_TAGS)
-def transformer_print(key, value=None, stack_offset=1, deferred=False, prefix=""):
-  return _mlperf_print(key=key, value=value, benchmark=TRANSFORMER,
-                       stack_offset=stack_offset, tag_set=TRANSFORMER_TAG_SET,
-                       deferred=deferred, root_dir=ROOT_DIR_TRANSFORMER,
-                       prefix=prefix)
-
-
-if __name__ == '__main__':
-  ncf_print(EVAL_ACCURACY, {'epoch': 7, 'accuracy': 43.7})
-  ncf_print(INPUT_SIZE, 1024)
diff --git a/compliance/mlperf_compliance/resnet_log_helper.py b/compliance/mlperf_compliance/resnet_log_helper.py
deleted file mode 100644
index eb7ff8e7a..000000000
--- a/compliance/mlperf_compliance/resnet_log_helper.py
+++ /dev/null
@@ -1,84 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Convenience functions for logging ResNet topology.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from mlperf_compliance import mlperf_log
-
-_STACK_OFFSET = 2
-
-def _get_shape(input_tensor):
-  return "({})".format(", ".join(
-      [str(i) for i in input_tensor.shape.as_list()[1:]]))
-
-
-def _in_out_shape(input_tensor, output_tensor):
-  return "{} -> {}".format( _get_shape(input_tensor), _get_shape(output_tensor))
-
-
-def log_max_pool(input_tensor, output_tensor):
-  mlperf_log.resnet_print(
-      key=mlperf_log.MODEL_HP_INITIAL_MAX_POOL, value=_in_out_shape(
-      input_tensor=input_tensor, output_tensor=output_tensor),
-      stack_offset=_STACK_OFFSET)
-
-
-def log_begin_block(input_tensor, block_type):
-  mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_BEGIN_BLOCK,
-                          value={"block_type": block_type},
-                          stack_offset=_STACK_OFFSET)
-  mlperf_log.resnet_print(
-      key=mlperf_log.MODEL_HP_RESNET_TOPOLOGY,
-      value=" Block Input: {}".format(_get_shape(input_tensor)),
-      stack_offset=_STACK_OFFSET)
-
-
-def log_end_block(output_tensor):
-  mlperf_log.resnet_print(
-      key=mlperf_log.MODEL_HP_END_BLOCK,
-      value=" Block Output: {}".format(_get_shape(output_tensor)),
-      stack_offset=_STACK_OFFSET)
-
-
-def log_projection(input_tensor, output_tensor):
-  mlperf_log.resnet_print(
-      key=mlperf_log.MODEL_HP_PROJECTION_SHORTCUT,
-      value=_in_out_shape(input_tensor, output_tensor),
-      stack_offset=_STACK_OFFSET)
-
-
-def log_conv2d(input_tensor, output_tensor, stride, filters, initializer,
-               use_bias):
-  mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_CONV2D_FIXED_PADDING,
-                          value=_in_out_shape(input_tensor, output_tensor),
-                          stack_offset=_STACK_OFFSET)
-  mlperf_log.resnet_print(
-      key=mlperf_log.MODEL_HP_CONV2D_FIXED_PADDING,
-      value={"stride": stride, "filters": filters, "initializer": initializer,
-             "use_bias": use_bias},
-      stack_offset=_STACK_OFFSET)
-
-
-def log_batch_norm(input_tensor, output_tensor, momentum, epsilon, center,
-                   scale, training):
-  assert _get_shape(input_tensor) == _get_shape(output_tensor)
-  mlperf_log.resnet_print(key=mlperf_log.MODEL_HP_BATCH_NORM, value={
-    "shape": _get_shape(input_tensor), "momentum": momentum, "epsilon": epsilon,
-    "center": center, "scale": scale, "training": training},
-                          stack_offset=_STACK_OFFSET)
diff --git a/compliance/mlperf_compliance/tags.py b/compliance/mlperf_compliance/tags.py
deleted file mode 100644
index 94bd4bb27..000000000
--- a/compliance/mlperf_compliance/tags.py
+++ /dev/null
@@ -1,620 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Master list of MLPerf tags to be logged for benchmark submissions.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from mlperf_compliance._gnmt_tags import *
-from mlperf_compliance._ncf_tags import *
-from mlperf_compliance._resnet_tags import *
-from mlperf_compliance._ssd_tags import *
-from mlperf_compliance._transformer_tags import *
-from mlperf_compliance._maskrcnn_tags import *
-
-# ==============================================================================
-# == Benchmarks ================================================================
-# ==============================================================================
-
-# rnn_translator
-GNMT = "gnmt"
-
-# reinforcement/
-MINIGO = "minigo"
-
-# recommendation/
-NCF = "ncf"
-
-# image_classification/
-RESNET = "resnet"
-
-# single_stage_detector/
-SSD = "ssd"
-
-# object_detection/
-MASKRCNN = "maskrcnn"
-
-# translation/
-TRANSFORMER = "transformer"
-
-# ==============================================================================
-# == Tags ======================================================================
-# ==============================================================================
-"""
-Tags may be used by all models, a subset of models, or only one model. A
-specification for which models require which tags can be found below the tag
-definitions.
-"""
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-# All models: Tags which should appear in absolutely every MLPerf model.
-# //////////////////////////////////////////////////////////////////////////////
-
-# This tag signals to start the timer. Emission of this tag need not be (and
-# generally will not be) the first part of a submission script. Rather, this
-# tag must be emitted prior to performing any work which the MLPerf rules
-# state must be timed. This tag is generally emitted directly before the first
-# step which invokes random number generation or the first step which must be
-# performed on the system under test. (Whichever comes first.) If clarification
-# is needed, please file an issue under:
-#   https://github.com/mlperf/policies
-RUN_START = "run_start"
-
-# This tag signals that a submission has reached the relevant stopping criteria,
-# and has completed all tasks which are performed in the reference. The wall
-# time for a submission will be computed as the difference between the time
-# when this tag is emitted and the time whe the RUN_START is emitted.
-RUN_STOP = "run_stop"
-
-# This tag should be emitted immediately before ending a run, and should be the
-# last tag emitted. This tag should indicate the completion of untimed post
-# processing work such as system specific cleanup.
-RUN_FINAL = "run_final"
-
-
-# Emit this tag in the place(s) where random seeds are set.
-RUN_SET_RANDOM_SEED = "run_set_random_seed"
-
-# Emit this tag when training data has been purged from volatile caches prior
-# to run start.
-RUN_CLEAR_CACHES = "run_clear_caches"
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-# Common Values: Constants which are expected to be reported across many models.
-#                These values are included for convenience.
-# //////////////////////////////////////////////////////////////////////////////
-BCE = "binary_cross_entropy"
-CCE = "categorical_cross_entropy"
-
-SGD = "stochastic_gradient_descent"
-
-# Some conventions distinguish between "vanilla" SGD and SGD with momentum
-# (where vanilla SGD would be the specific case of momentum=0)
-SGD_WITH_MOMENTUM = "stochastic_gradient_descent_with_momentum"
-
-ADAM = "adam"
-LAZY_ADAM = "lazy_adam"
-
-TRUNCATED_NORMAL = "truncated_normal"
-
-RELU = "relu"
-
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-# Preprocessing: Tags for generic preprocessing steps
-# //////////////////////////////////////////////////////////////////////////////
-
-# The number of training examples in a single epoch
-PREPROC_NUM_TRAIN_EXAMPLES = "preproc_num_train_examples"
-
-# The number of evaluation examples in a single epoch
-PREPROC_NUM_EVAL_EXAMPLES = "preproc_num_eval_examples"
-
-# This tag is used to declare what part of code tokenizes the training data.
-PREPROC_TOKENIZE_TRAINING = "preproc_tokenize_training"
-
-# This tag is used to declare what part of code tokenizes the evaluation data.
-PREPROC_TOKENIZE_EVAL = "preproc_tokenize_eval"
-
-# The vocabulary size used for tokenization.
-PREPROC_VOCAB_SIZE = "preproc_vocab_size"
-
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-# Input: Tags for the timed portion of the data input pipeline
-# //////////////////////////////////////////////////////////////////////////////
-
-# The number of examples in the training portion of the data pipeline. Generally
-# this should match PREPROC_NUM_TRAIN_EXAMPLES. If it does not (for instance
-# if certain examples are dropped in compliance with MLPerf rules), the
-# call which declares this tag is a good place for a comment stating why the
-# disparity is expected.
-INPUT_SIZE = "input_size"
-
-# The size of a training minibatch size. If this value is variable, please emit
-# "-1" and then log an implementation specific characterization of the batch
-# size which is a reasonable analog to the reference. (For instance log that
-# all but the last batch has size 64, and the last batch is a partial batch)
-INPUT_BATCH_SIZE = "input_batch_size"
-
-# This tag indicates where the location of the code which defines the order in
-# which training examples are traversed. It is not necessary to describe the
-# method in the tag emission (though comments are always welcome). Rather, this
-# should simply provide a good starting point to an interested party.
-INPUT_ORDER = "input_order"
-
-# The shard size (in items) when shuffling in the input pipeline.
-INPUT_SHARD = "input_shard"
-
-# The number of samples iver which BN stats are computed for normalization during training
-INPUT_BN_SPAN = "input_bn_span"
-
-
-# --------------------------------------
-# -- Data Augmentation and Alteration --
-# --------------------------------------
-
-# ResNet random cropping
-INPUT_CENTRAL_CROP = "input_central_crop"
-
-INPUT_CROP_USES_BBOXES = "input_crop_uses_bboxes"
-
-INPUT_DISTORTED_CROP_MIN_OBJ_COV = "input_distorted_crop_min_object_covered"
-INPUT_DISTORTED_CROP_RATIO_RANGE = "input_distorted_crop_aspect_ratio_range"
-INPUT_DISTORTED_CROP_AREA_RANGE = "input_distorted_crop_area_range"
-INPUT_DISTORTED_CROP_MAX_ATTEMPTS = "input_distorted_crop_max_attempts"
-
-INPUT_MEAN_SUBTRACTION = "input_mean_subtraction"
-
-# Random flip of an image for data augmentation
-INPUT_RANDOM_FLIP = "input_random_flip"
-
-INPUT_RESIZE = "input_resize"
-INPUT_RESIZE_ASPECT_PRESERVING = "input_resize_aspect_preserving"
-
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-# Opt: Tags for declaring optimizer specific information. Submissions should
-#      declare and log explicit values rather than relying on defaults.
-# //////////////////////////////////////////////////////////////////////////////
-
-# The name of the optimizer used. (SGD, Adam, etc.)
-OPT_NAME = "opt_name"
-
-OPT_LR = "opt_learning_rate"
-OPT_MOMENTUM = "opt_momentum"
-
-OPT_WEIGHT_DECAY = "opt_weight_decay"
-
-# beta1, beta2, and epsilon are optimizer hyperparameters associated with the
-# Adam optimizer and its variants (e.g. LazyAdam).
-OPT_HP_ADAM_BETA1 = "opt_hp_Adam_beta1"
-OPT_HP_ADAM_BETA2 = "opt_hp_Adam_beta2"
-OPT_HP_ADAM_EPSILON = "opt_hp_Adam_epsilon"
-
-# The number of warm-up steps (SGD).
-OPT_LR_WARMUP_STEPS = "opt_learning_rate_warmup_steps"
-
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-#  Train: Tags for control flow during model training.
-# //////////////////////////////////////////////////////////////////////////////
-
-# This tag is emitted when a model first enters its training loop. This is not
-# necessarily when it begins to apply gradients; rather, it should be placed at
-# a location which logically partitions the submission code.
-TRAIN_LOOP = "train_loop"
-
-# The current epoch as said epoch begins training.
-TRAIN_EPOCH = "train_epoch"
-
-# This tag is used to indicate approximately where checkpoints are written. Some
-# frameworks abstract away checkpoint saving; in such cases simply choose a
-# logical place in the code which signals that the framework has been instructed
-# to save checkpoints, along with an explanatory comment.
-TRAIN_CHECKPOINT = "train_checkpoint"
-
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-#  Eval: Tags for control flow during model evaluation.
-# //////////////////////////////////////////////////////////////////////////////
-
-# This tag should be emitted whenever the submission begins an evaluation pass
-# for a given set of weights.
-EVAL_START = "eval_start"
-
-# The number of examples on which evaluation is performed.
-EVAL_SIZE = "eval_size"
-
-# The target quality at which the model may stop training.
-EVAL_TARGET = "eval_target"
-
-# The observed accuracy of the model at a given epoch.
-EVAL_ACCURACY = "eval_accuracy"
-
-# This tag should be emitted whenever the submission ends an evaluation pass
-# for a given set of weights.
-EVAL_STOP = "eval_stop"
-
-
-# The observed accuracy of the model at a given iteration. This is only for
-# models which evaluate at certain iterations instead of epochs.
-EVAL_ITERATION_ACCURACY = "eval_iteration_accuracy"
-
-# \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
-#  Model: Tags for logging topology specific information.
-# //////////////////////////////////////////////////////////////////////////////
-
-# The loss function (cross entropy, squared error, etc.) used by the model. For
-# more exotic loss functions such as those encountered in object detection
-# models, additional benchmark specific subcomponents should also be logged.
-MODEL_HP_LOSS_FN = "model_hp_loss_fn"
-
-MODEL_HP_INITIAL_SHAPE = "model_hp_initial_shape"
-MODEL_HP_FINAL_SHAPE = "model_hp_final_shape"
-
-MODEL_L2_REGULARIZATION = "model_l2_regularization"
-MODEL_EXCLUDE_BN_FROM_L2 = "model_exclude_bn_from_l2"
-
-MODEL_HP_RELU = "model_hp_relu"
-MODEL_HP_CONV2D_FIXED_PADDING = "model_hp_conv2d_fixed_padding"
-MODEL_HP_BATCH_NORM = "model_hp_batch_norm"
-MODEL_HP_DENSE = "model_hp_dense"
-
-
-# ==============================================================================
-# == Stdout tags ===============================================================
-# ==============================================================================
-
-# These tags are always logged to stdout. The rest will be logged to a file if
-# one is available.
-STDOUT_TAG_SET = {
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-}
-
-
-# ==============================================================================
-# == Benchmark tag sets ========================================================
-# ==============================================================================
-ALL_USED_TAGS = set()
-
-GNMT_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_SET_RANDOM_SEED,
-    RUN_CLEAR_CACHES,
-
-    PREPROC_VOCAB_SIZE,
-    PREPROC_TOKENIZE_TRAINING,
-    PREPROC_TOKENIZE_EVAL,
-    PREPROC_NUM_TRAIN_EXAMPLES,
-    PREPROC_NUM_EVAL_EXAMPLES,
-
-    INPUT_SIZE,
-    INPUT_BATCH_SIZE,
-    INPUT_ORDER,
-    INPUT_SHARD,
-
-    OPT_NAME,
-    OPT_LR,
-    OPT_LR_WARMUP_STEPS,
-    OPT_HP_ADAM_BETA1,
-    OPT_HP_ADAM_BETA2,
-    OPT_HP_ADAM_EPSILON,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-    TRAIN_CHECKPOINT,
-    TRAIN_HP_MAX_SEQ_LEN,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-    EVAL_HP_BEAM_SIZE,
-    EVAL_HP_MAX_SEQ_LEN,
-    EVAL_HP_LEN_NORM_CONST,
-    EVAL_HP_LEN_NORM_FACTOR,
-    EVAL_HP_COV_PENALTY_FACTOR,
-
-    MODEL_HP_LOSS_FN,
-    MODEL_HP_LOSS_SMOOTHING,
-    MODEL_HP_NUM_LAYERS,
-    MODEL_HP_HIDDEN_SIZE,
-    MODEL_HP_DROPOUT
-)
-
-MASKRCNN_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_CLEAR_CACHES,
-
-    INPUT_SIZE,
-    INPUT_BATCH_SIZE,
-    GLOBAL_BATCH_SIZE,
-    INPUT_ORDER,
-    INPUT_SHARD,
-
-    BACKBONE,
-    NMS_THRESHOLD,
-    NMS_MAX_DETECTIONS,
-
-    OPT_NAME,
-    OPT_LR,
-    OPT_LR_WARMUP_STEPS,
-    OPT_MOMENTUM,
-    OPT_WEIGHT_DECAY,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-    INPUT_MEAN_SUBTRACTION,
-    INPUT_NORMALIZATION_STD,
-    INPUT_RESIZE,
-    INPUT_RESIZE_ASPECT_PRESERVING,
-    MIN_IMAGE_SIZE,
-    MAX_IMAGE_SIZE,
-    RUN_SET_RANDOM_SEED,
-    INPUT_RANDOM_FLIP,
-    RANDOM_FLIP_PROBABILITY,
-    FG_IOU_THRESHOLD,
-    BG_IOU_THRESHOLD,
-    RPN_PRE_NMS_TOP_N_TRAIN,
-    RPN_PRE_NMS_TOP_N_TEST,
-    RPN_POST_NMS_TOP_N_TRAIN,
-    RPN_POST_NMS_TOP_N_TEST,
-    BATCH_SIZE_TEST,
-    ASPECT_RATIOS,
-)
-
-MINIGO_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_SET_RANDOM_SEED,
-    RUN_CLEAR_CACHES,
-
-    INPUT_SHARD,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-)
-
-NCF_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_CLEAR_CACHES,
-
-    PREPROC_HP_MIN_RATINGS,
-    PREPROC_HP_NUM_EVAL,
-    PREPROC_HP_SAMPLE_EVAL_REPLACEMENT,
-
-    INPUT_SIZE,
-    INPUT_BATCH_SIZE,
-    INPUT_ORDER,
-    INPUT_SHARD,
-    INPUT_HP_NUM_NEG,
-    INPUT_HP_SAMPLE_TRAIN_REPLACEMENT,
-    INPUT_STEP_TRAIN_NEG_GEN,
-    INPUT_STEP_EVAL_NEG_GEN,
-
-    OPT_NAME,
-    OPT_LR,
-    OPT_HP_ADAM_BETA1,
-    OPT_HP_ADAM_BETA2,
-    OPT_HP_ADAM_EPSILON,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-    EVAL_HP_NUM_USERS,
-    EVAL_HP_NUM_NEG,
-
-    MODEL_HP_LOSS_FN,
-    MODEL_HP_MF_DIM,
-    MODEL_HP_MLP_LAYER_SIZES,
-)
-
-RESNET_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_SET_RANDOM_SEED,
-    RUN_CLEAR_CACHES,
-
-    PREPROC_NUM_TRAIN_EXAMPLES,
-    PREPROC_NUM_EVAL_EXAMPLES,
-
-    INPUT_SIZE,
-    INPUT_BATCH_SIZE,
-    INPUT_ORDER,
-    INPUT_SHARD,
-    INPUT_CENTRAL_CROP,
-    INPUT_CROP_USES_BBOXES,
-    INPUT_DISTORTED_CROP_MIN_OBJ_COV,
-    INPUT_DISTORTED_CROP_RATIO_RANGE,
-    INPUT_DISTORTED_CROP_AREA_RANGE,
-    INPUT_DISTORTED_CROP_MAX_ATTEMPTS,
-    INPUT_MEAN_SUBTRACTION,
-    INPUT_RANDOM_FLIP,
-    INPUT_RESIZE,
-    INPUT_RESIZE_ASPECT_PRESERVING,
-    INPUT_BN_SPAN,
-
-    OPT_NAME,
-    OPT_LR,
-    OPT_LR_WARMUP_STEPS,
-    OPT_MOMENTUM,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-    EVAL_EPOCH_OFFSET,
-
-    MODEL_HP_LOSS_FN,
-    MODEL_L2_REGULARIZATION,
-    MODEL_EXCLUDE_BN_FROM_L2,
-
-    MODEL_HP_INITIAL_SHAPE,
-    MODEL_HP_FINAL_SHAPE,
-    MODEL_HP_INITIAL_MAX_POOL,
-    MODEL_HP_BEGIN_BLOCK,
-    MODEL_HP_END_BLOCK,
-    MODEL_HP_BLOCK_TYPE,
-    MODEL_HP_PROJECTION_SHORTCUT,
-    MODEL_HP_SHORTCUT_ADD,
-    MODEL_HP_RELU,
-    MODEL_HP_CONV2D_FIXED_PADDING,
-    MODEL_HP_BATCH_NORM,
-    MODEL_HP_DENSE,
-    MODEL_HP_RESNET_TOPOLOGY,
-)
-
-SSD_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_CLEAR_CACHES,
-
-    INPUT_SIZE,
-    INPUT_BATCH_SIZE,
-    INPUT_ORDER,
-    INPUT_SHARD,
-    INPUT_BN_SPAN,
-
-    BACKBONE,
-    FEATURE_SIZES,
-    STEPS,
-    SCALES,
-    ASPECT_RATIOS,
-    NUM_DEFAULTS_PER_CELL,
-    LOC_CONF_OUT_CHANNELS,
-    NUM_DEFAULTS,
-    NMS_THRESHOLD,
-    NMS_MAX_DETECTIONS,
-
-    NUM_CROPPING_ITERATIONS,
-    RANDOM_FLIP_PROBABILITY,
-    DATA_NORMALIZATION_MEAN,
-    DATA_NORMALIZATION_STD,
-
-    OPT_NAME,
-    OPT_LR,
-    OPT_MOMENTUM,
-    OPT_WEIGHT_DECAY,
-    OPT_LR_WARMUP_STEPS,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_SIZE,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-    EVAL_ITERATION_ACCURACY,
-)
-
-TRANSFORMER_TAGS = (
-    RUN_START,
-    RUN_STOP,
-    RUN_FINAL,
-    RUN_SET_RANDOM_SEED,
-    RUN_CLEAR_CACHES,
-
-    PREPROC_NUM_TRAIN_EXAMPLES,
-    PREPROC_NUM_EVAL_EXAMPLES,
-    PREPROC_TOKENIZE_TRAINING,
-    PREPROC_TOKENIZE_EVAL,
-    PREPROC_VOCAB_SIZE,
-
-    INPUT_BATCH_SIZE,
-    INPUT_MAX_LENGTH,
-    INPUT_ORDER,
-    INPUT_SHARD,
-
-    OPT_NAME,
-    OPT_LR,
-    OPT_LR_WARMUP_STEPS,
-    OPT_HP_ADAM_BETA1,
-    OPT_HP_ADAM_BETA2,
-    OPT_HP_ADAM_EPSILON,
-
-    TRAIN_LOOP,
-    TRAIN_EPOCH,
-
-    EVAL_START,
-    EVAL_TARGET,
-    EVAL_ACCURACY,
-    EVAL_STOP,
-
-    MODEL_HP_INITIALIZER_GAIN,
-    MODEL_HP_VOCAB_SIZE,
-    MODEL_HP_NUM_HIDDEN_LAYERS,
-    MODEL_HP_EMBEDDING_SHARED_WEIGHTS,
-    MODEL_HP_ATTENTION_DENSE,
-    MODEL_HP_ATTENTION_DROPOUT,
-    MODEL_HP_FFN_OUTPUT_DENSE,
-    MODEL_HP_FFN_FILTER_DENSE,
-    MODEL_HP_RELU_DROPOUT,
-    MODEL_HP_LAYER_POSTPROCESS_DROPOUT,
-    MODEL_HP_NORM,
-    MODEL_HP_SEQ_BEAM_SEARCH,
-)
-
-ALL_USED_TAGS.update(GNMT_TAGS)
-ALL_USED_TAGS.update(MASKRCNN_TAGS)
-ALL_USED_TAGS.update(NCF_TAGS)
-ALL_USED_TAGS.update(RESNET_TAGS)
-ALL_USED_TAGS.update(SSD_TAGS)
-ALL_USED_TAGS.update(TRANSFORMER_TAGS)
diff --git a/compliance/mlperf_compliance/tf_mlperf_log.py b/compliance/mlperf_compliance/tf_mlperf_log.py
deleted file mode 100644
index e081c69ab..000000000
--- a/compliance/mlperf_compliance/tf_mlperf_log.py
+++ /dev/null
@@ -1,95 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Convenience function for extracting the values for logging calls.
-
-Because TensorFlow generally defers computation of values to a session run call,
-it is impractical to log the values of tensors when they are defined. Instead,
-the definition of a tensor is logged as normal using the log function in
-mlperf_log.py and a tf.print statement helper function can be used to report
-the relevant values as they are computed.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import uuid
-
-import tensorflow as tf
-
-
-def log_deferred(op, log_id, every_n=1, first_n=None):
-  """Helper method inserting compliance logging ops.
-
-  Note: This helper is not guaranteed to be efficient, as it will insert ops
-        and control dependencies. If this proves to be a bottleneck, submitters
-        may wish to consider other methods such as extracting values from an
-        .events file.
-
-  Args:
-    op: A tf op to be printed.
-    log_id: a uuid provided by the logger in mlperf_log.py
-    every_n: If repeat is True, with what frequency should the input op be '
-             logged. If repeat is False, this argument is ignored.
-    first_n: Only log this many values. This arg does not interact with every_n.
-             The first_n refers to the first n that would have been logged.
-  """
-
-  prefix = ":::MLPv0.5.0 [{}]".format(log_id)
-  if not first_n is not None and first_n == 1:
-    return tf.Print(op, [tf.timestamp(), op], message=prefix, first_n=1)
-
-  counter = tf.Variable(tf.zeros(shape=(), dtype=tf.int32) - 1,
-                        aggregation=tf.VariableAggregation.MEAN)
-  increment = tf.assign_add(counter, 1, use_locking=True)
-  return tf.cond(
-      tf.equal(tf.mod(increment, every_n), 0),
-      lambda :tf.Print(op, [tf.timestamp(), op], message=prefix,
-                       first_n=first_n),
-      lambda :op
-  )
-
-
-def sum_metric(tensor, name):
-  sum_var = tf.Variable(
-    initial_value=tf.zeros(shape=(), dtype=tensor.dtype),
-    trainable=False,
-    collections=[
-      tf.GraphKeys.LOCAL_VARIABLES,
-      tf.GraphKeys.METRIC_VARIABLES,
-    ],
-    name="{}_total".format(name),
-    aggregation=tf.VariableAggregation.SUM
-  )
-
-  update_op = tf.identity(tf.assign_add(sum_var, tensor))
-  return tf.identity(sum_var, name=name), update_op
-
-
-def _example():
-  for kwargs in [dict(first_n=1), dict(), dict(every_n=2),
-                 dict(first_n=2, every_n=2)]:
-    op = tf.assign_add(tf.Variable(tf.zeros(shape=(), dtype=tf.int32) - 1), 1)
-    op = log_deferred(op, str(uuid.uuid4()), **kwargs)
-    init = [tf.local_variables_initializer(), tf.global_variables_initializer()]
-    print("-" * 5)
-    with tf.Session().as_default() as sess:
-      sess.run(init)
-      for _ in range(6):
-        sess.run(op)
-
-
-if __name__ == "__main__":
-  _example()
diff --git a/compliance/setup.py b/compliance/setup.py
deleted file mode 100644
index b31348463..000000000
--- a/compliance/setup.py
+++ /dev/null
@@ -1,38 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-import setuptools
-
-with open("README.md", "r") as f:
-  long_description = f.read()
-
-setuptools.setup(
-    name="mlperf_compliance",
-    version="0.1.0",
-    author="MLPerf.org",
-    author_email="mlperf@googlegroups.com",
-    description="MLPerf compliance tools.",
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    url="https://github.com/mlperf/training/tree/master/compliance",
-    packages=setuptools.find_namespace_packages(include=['mllog*']),
-    classifiers=[
-      "Programming Language :: Python :: 2",
-      "Programming Language :: Python :: 3",
-      "License :: OSI Approved :: Apache Software License",
-      "Operating System :: OS Independent",
-    ],
-    license='Apache 2.0',
-)
diff --git a/compliance/verify_submission/README.md b/compliance/verify_submission/README.md
deleted file mode 100644
index 5113a3f8e..000000000
--- a/compliance/verify_submission/README.md
+++ /dev/null
@@ -1,47 +0,0 @@
-# mlperf-submission-helper
-
-## Key generation
-
-If you need to encrypt your submission, run the followings to generate a key pair.
-
-```
-ssh-keygen -t rsa -b 1024 -f ${KEY_NAME} -N ''
-```
-where `${KEY_NAME}` is the name of your key files. You can give any valid file name.
-
-In the generated files, the private key should be named `${KEY_NAME}`, the public key is named `${KEY_NAME}.pub`.
-
-## Install
-
-```
-git clone --recurse-submodules https://github.com/mlperf/training.git
-cd training/compliance/verify_submission
-
-# If you need encryption/decryption:
-pip install -r crypto_requirements.txt
-```
-
-## Run
-
-- Basic verification (no encryption/decryption):
-
-```
-python mlperf_submission_helper/verify_submission.py ${SUBMISSION_ROOT}
-```
-where `${SUBMISSION_ROOT}` is the root directory of your submission.
-
-- Verification with encryption (the submission will be verified and then encrypted and saved at a new directory):
-
-```
-python mlperf_submission_helper/verify_submission.py \
---encrypt-key ${PUBLIC_KEY} --encrypt-out ${ENCRYPT_OUT} ${SUBMISSION_ROOT}
-```
-where `${PUBLIC_KEY}` is the path to your public key, `${ENCRYPT_OUT}` is the path to encrypted submission (it should be a new directory).
-
-- Verification with decryption (the submission will be decrypted and saved at a new directory, and then the decrypted submission will be verified):
-
-```
-python mlperf_submission_helper/verify_submission.py \
---decrypt-key ${PRIVATE_KEY} --decrypt-out ${DECRYPT_OUT} ${SUBMISSION_ROOT}
-```
-where `${PRIVATE_KEY}` is the path to your private key, `${DECRYPT_OUT}` is the path to decrypted submission (it should be a new directory).
diff --git a/compliance/verify_submission/crypto_requirements.txt b/compliance/verify_submission/crypto_requirements.txt
deleted file mode 100644
index c3466db77..000000000
--- a/compliance/verify_submission/crypto_requirements.txt
+++ /dev/null
@@ -1,1 +0,0 @@
-pycryptodomex==v3.7.0
diff --git a/compliance/verify_submission/examples/example_report.txt b/compliance/verify_submission/examples/example_report.txt
deleted file mode 100644
index 098f719f0..000000000
--- a/compliance/verify_submission/examples/example_report.txt
+++ /dev/null
@@ -1,165 +0,0 @@
-MLPERF SUBMISSION REPORT
-========================
-
-
-SUMMARY
--------
-
-Passed checks: 135
-Failed checks: 0
-Errors: 0
-Note: the Errors indicate certain checks being skipped or early terminated due to errors. The errors are likely caused by failed checks above.
-
-PASSED CHECKS
--------------
-
-Path exists: ../../example_submission/results
-Path exists: ../../example_submission/code
-Path exists: ../../example_submission/submission.json
-../../example_submission/code/resnet name is in ['resnet', 'ssd', 'maskrcnn', 'transformer', 'gnmt', 'ncf', 'minigo', 'shared'].
-Path exists: ../../example_submission/code/resnet/README.md
-Path exists: ../../example_submission/code/resnet/preproc_dataset.sh
-../../example_submission/code/ncf name is in ['resnet', 'ssd', 'maskrcnn', 'transformer', 'gnmt', 'ncf', 'minigo', 'shared'].
-Path exists: ../../example_submission/code/ncf/README.md
-Path exists: ../../example_submission/code/ncf/preproc_dataset.sh
-Path exists: ../../example_submission/results/zpu1x/entry.json
-../../example_submission/results/zpu1x/resnet name is in ['resnet', 'ssd', 'maskrcnn', 'transformer', 'gnmt', 'ncf', 'minigo'].
-Path exists: ../../example_submission/code/resnet/setup_zpu1x.sh
-Path exists: ../../example_submission/code/resnet/run_and_time_zpu1x.sh
-Path exists: ../../example_submission/results/zpu1x/resnet/result_0.txt
-Path exists: ../../example_submission/results/zpu1x/resnet/result_1.txt
-Path exists: ../../example_submission/results/zpu1x/resnet/result_2.txt
-Path exists: ../../example_submission/results/zpu1x/resnet/result_3.txt
-Path exists: ../../example_submission/results/zpu1x/resnet/result_4.txt
-../../example_submission/results/zpu1x/ncf name is in ['resnet', 'ssd', 'maskrcnn', 'transformer', 'gnmt', 'ncf', 'minigo'].
-Path exists: ../../example_submission/code/ncf/setup_zpu1x.sh
-Path exists: ../../example_submission/code/ncf/run_and_time_zpu1x.sh
-Path exists: ../../example_submission/results/zpu1x/ncf/result_0.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_1.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_2.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_3.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_4.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_5.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_6.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_7.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_8.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_9.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_10.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_11.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_12.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_13.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_14.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_15.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_16.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_17.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_18.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_19.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_20.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_21.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_22.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_23.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_24.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_25.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_26.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_27.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_28.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_29.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_30.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_31.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_32.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_33.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_34.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_35.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_36.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_37.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_38.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_39.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_40.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_41.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_42.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_43.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_44.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_45.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_46.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_47.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_48.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_49.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_50.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_51.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_52.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_53.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_54.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_55.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_56.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_57.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_58.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_59.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_60.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_61.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_62.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_63.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_64.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_65.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_66.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_67.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_68.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_69.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_70.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_71.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_72.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_73.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_74.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_75.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_76.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_77.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_78.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_79.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_80.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_81.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_82.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_83.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_84.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_85.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_86.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_87.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_88.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_89.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_90.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_91.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_92.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_93.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_94.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_95.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_96.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_97.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_98.txt
-Path exists: ../../example_submission/results/zpu1x/ncf/result_99.txt
-Path exists: ../../example_submission/results/zpu8x/entry.json
-../../example_submission/results/zpu8x/resnet name is in ['resnet', 'ssd', 'maskrcnn', 'transformer', 'gnmt', 'ncf', 'minigo'].
-Path exists: ../../example_submission/code/resnet/setup_zpu8x.sh
-Path exists: ../../example_submission/code/resnet/run_and_time_zpu8x.sh
-Path exists: ../../example_submission/results/zpu8x/resnet/result_0.txt
-Path exists: ../../example_submission/results/zpu8x/resnet/result_1.txt
-Path exists: ../../example_submission/results/zpu8x/resnet/result_2.txt
-Path exists: ../../example_submission/results/zpu8x/resnet/result_3.txt
-Path exists: ../../example_submission/results/zpu8x/resnet/result_4.txt
-Keys in submission metadata match expected.
-Keys in entry zpu1x metadata match expected.
-Keys in entry zpu1x node metadata match expected.
-Keys in entry zpu8x metadata match expected.
-Keys in entry zpu8x node metadata match expected.
-
-FAILED CHECKS
--------------
-
-
-ERRORS
-------
-
-
-RESULTS
--------
-
-org,division,status,hardware,framework,power,notes,resnet,ssd,maskrcnn,transformer,gnmt,ncf,minigo
-Test Inc.,open,on-premise,1 ZPU,TensorFlow 1.12,https://mlperf.org/,ZPUs are the best,12.0,-,-,-,-,44.2916666667,-
-Test Inc.,open,on-premise,1 ZPU,TensorFlow 1.12,https://mlperf.org/,ZPUs are the best,7.0,-,-,-,-,-,-
diff --git a/compliance/verify_submission/mlperf_submission_helper/checks.py b/compliance/verify_submission/mlperf_submission_helper/checks.py
deleted file mode 100644
index 3a514630d..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/checks.py
+++ /dev/null
@@ -1,309 +0,0 @@
-"""Checks to run against a submission."""
-from __future__ import print_function
-
-import json
-import os
-
-import constants
-import mlp_compliance.mlp_compliance as mlp_compliance
-import report as subm_report
-
-INFINITE_TIME = 9999999999.99
-
-
-class SubmissionChecks(object):
-  """Submission checks."""
-
-  def __init__(self):
-    self.report = subm_report.SubmissionReport()
-    self.submission_meta = {}
-    self.result_meta = {}
-    self.result_entry_meta = {}
-
-  def verify_dirs_and_files(self, root_dir):
-    self.verify_root_dir(root_dir)
-    self.verify_code_dir(root_dir)
-    self.verify_results_dir(root_dir)
-
-  def verify_metadata(self):
-    """Verifies the metadata.
-
-        Must be called after verify_dirs_and_files()
-    """
-    self.verify_submission_metadata()
-    self.verify_result_entry_metadata()
-
-  def exists(self, path, is_dir=False):
-    exists_fn = os.path.isdir if is_dir else os.path.isfile
-    if exists_fn(path):
-      self.report.add_passed_check('Path exists: {}'.format(path))
-    else:
-      self.report.add_failed_check('Path not found: {}'.format(path))
-
-  def name_in(self, path, ref_list):
-    basename = os.path.basename(path)
-    if basename in ref_list:
-      self.report.add_passed_check('{} name is in {}.'.format(path, ref_list))
-    else:
-      self.report.add_failed_check('{} name not in {}.'.format(path, ref_list))
-
-  def keys_match(self, keys, ref_keys, context=''):
-    different_keys = set(keys).difference(set(ref_keys))
-    if different_keys:
-      self.report.add_failed_check('Keys in {} do not match expected: '.format(
-          context) + 'unmatched keys: {}'.format(list(different_keys)))
-    else:
-      self.report.add_passed_check('Keys in {} match expected.'.format(context))
-
-  def verify_root_dir(self, root_dir):
-    result_dir = os.path.join(root_dir, 'results')
-    code_dir = os.path.join(root_dir, 'code')
-    submission_meta_file = os.path.join(root_dir, 'submission.json')
-
-    self.exists(result_dir, is_dir=True)
-    self.exists(code_dir, is_dir=True)
-    self.exists(submission_meta_file, is_dir=False)
-
-    try:
-      with open(submission_meta_file) as f:
-        self.submission_meta = json.load(f)
-    except Exception as e:
-      self.report.add_error('Unable to parse submission meatadata: {}'.format(
-          str(e)))
-
-  def verify_code_dir(self, root_dir):
-    code_root_dir = os.path.join(root_dir, 'code')
-    try:
-      for code_name in os.listdir(code_root_dir):
-        code_dir = os.path.join(code_root_dir, code_name)
-        if not os.path.isdir(code_dir):
-          continue
-        self.name_in(code_dir, constants.BENCHMARK_NAMES + ['shared'])
-        if code_name in constants.BENCHMARK_NAMES:
-          self.exists(os.path.join(code_dir, 'README.md'))
-          self.exists(os.path.join(code_dir, 'preproc_dataset.sh'))
-    except Exception as e:
-      self.report.add_error('Unable to verify code dir: {}'.format(str(e)))
-
-  def verify_results_dir(self, root_dir):
-    code_root_dir = os.path.join(root_dir, 'code')
-    result_root_dir = os.path.join(root_dir, 'results')
-    try:
-      for entry_name in os.listdir(result_root_dir):
-        entry_dir = os.path.join(result_root_dir, entry_name)
-        if not os.path.isdir(entry_dir):
-          continue
-        entry_meta_file = os.path.join(entry_dir, 'entry.json')
-        try:
-          with open(entry_meta_file) as f:
-            self.result_entry_meta[entry_name] = json.load(f)
-        except Exception as e:
-          self.report.add_error(
-              'Unable to parse result entry metadata: {}'.format(str(e)))
-        self.exists(entry_meta_file)
-        for result_name in os.listdir(entry_dir):
-          result_dir = os.path.join(entry_dir, result_name)
-          if not os.path.isdir(result_dir):
-            continue
-          self.name_in(result_dir, constants.BENCHMARK_NAMES)
-          self.exists(
-              os.path.join(code_root_dir, result_name,
-                           'setup_' + entry_name + '.sh'))
-          self.exists(
-              os.path.join(code_root_dir, result_name,
-                           'run_and_time_' + entry_name + '.sh'))
-          result_num = constants.REQUIRED_RESULT_NUM.get(result_name, 0)
-          for i in range(result_num):
-            log_file_name = 'result_' + str(i) + '.txt'
-            self.exists(os.path.join(result_dir, log_file_name))
-            self.result_meta.setdefault(entry_name, {})
-            self.result_meta[entry_name].setdefault(
-                result_name, [None for j in range(result_num)])
-            division = self.result_entry_meta[entry_name].get('division')
-            log_path = os.path.join(result_dir, log_file_name)
-            dt, start_time = self.verify_and_extract_time(log_path,
-                                                          division,
-                                                          result_name)
-            self._add_result(self.result_meta[entry_name][result_name],
-                             i,
-                             dt,
-                             start_time)
-    except Exception as e:
-      self.report.add_error('Unable to verify results dir: {}'.format(str(e)))
-
-  def _add_result(self, dict_entry, entry, dt, start_time):
-    """Adds a result to the dictionary.
-
-    Args:
-      dict_entry: main dict to add entry
-      entry: slot for this entry (likely an integer)
-      dt: the timing for the entry
-      start_time: when the entry started unix time float
-    """
-    time_entry = {}
-    time_entry['dt'] = dt
-    time_entry['start_time'] = start_time
-    dict_entry[entry] = time_entry
-
-  def _sorted_results(self, results_dicts):
-    """Sorts dict of results based on log start_time.
-
-    Sorts the results and returns an array with only the values but sorted
-    by oldest value first.value
-
-    Args:
-      results_dicts: List of result dicts
-
-    Returns:
-      List of only the time but sorted oldest first.
-    """
-    print('results dicts:', results_dicts)
-    sorted_dict = sorted(results_dicts, key=lambda k: k['start_time'])
-    results = []
-    for entry in sorted_dict:
-      results.append(entry['dt'])
-    return results
-
-  def verify_submission_metadata(self):
-    subm_meta_keys = self.submission_meta.keys()
-    different_keys = set(subm_meta_keys).difference(
-        set(constants.SUBM_META_PROPS))
-    if different_keys:
-      self.report.add_failed_check(
-          'Keys in submission metadata do not match expected: ' +
-          'unmatched keys: {}'.format(list(different_keys)))
-    else:
-      self.report.add_passed_check(
-          'Keys in submission metadata match expected.')
-
-  def verify_result_entry_metadata(self):
-    for entry_name in self.result_entry_meta:
-      entry_meta = self.result_entry_meta[entry_name]
-      entry_meta_keys = entry_meta.keys()
-      self.keys_match(
-          entry_meta_keys,
-          constants.ENTRY_META_PROPS,
-          context='entry {} metadata'.format(entry_name))
-      try:
-        for node_meta in entry_meta['nodes']:
-          node_meta_keys = node_meta.keys()
-          self.keys_match(
-              node_meta_keys,
-              constants.NODE_META_PROPS,
-              context='entry {} node metadata'.format(entry_name))
-      except Exception as e:
-        self.report.add_error(
-            'Unable to verify node metadata for entry {}: {}'.format(
-                entry_name, str(e)))
-
-  def compile_results(self):
-    results = {}
-    for entry_name in self.result_meta:
-      results.setdefault(entry_name, {})
-      for key in constants.RESULT_SUBM_META_COLUMNS:
-        results[entry_name][key] = self.submission_meta[key]
-      for key in constants.RESULT_ENTRY_META_COLUMNS:
-        results[entry_name][key] = self.result_entry_meta[entry_name][key]
-      for benchmark_name in constants.BENCHMARK_NAMES:
-        benchmark_results = self.result_meta[entry_name].get(
-            benchmark_name, None)
-        if not benchmark_results:
-          results[entry_name][benchmark_name] = None
-          continue
-        if not all(benchmark_results):
-          self.report.add_error('Benchmark results contain None values. ' +
-                                'entry: {}, benchmark name: {}'.format(
-                                    entry_name, benchmark_name))
-          results[entry_name][benchmark_name] = None
-          continue
-        # Turns dict into list or results oldest first
-        benchmark_results = self._sorted_results(benchmark_results)
-        # special treatment for the NCF results
-        if benchmark_name == 'ncf':
-          possible_results = benchmark_results
-          benchmark_results = []
-          for pr in possible_results:
-            # Skips results that were failures to converge, which are set to
-            # INFINITE_TIME
-            if pr != INFINITE_TIME:
-              benchmark_results.append(pr)
-            # Stops after it finds 50 results, which may not be the first
-            # 50 by date unless they are sorted earlier.
-            if len(benchmark_results) == 50:
-              break
-          if len(benchmark_results) != 50:
-            raise Exception('NCF does not have 50 good results')
-          if len(possible_results) != 100:
-            raise Exception('NCF does not have 100 good results:{}'.format(
-                len(possible_results)))
-
-        benchmark_results = sorted(benchmark_results)
-        print('benchmark_name:{}|{} results{}'.format(benchmark_name,
-                                                      entry_name,
-                                                      benchmark_results))
-        del benchmark_results[0]
-        del benchmark_results[-1]
-        result_val = (
-            float(sum(benchmark_results)) / len(benchmark_results) /
-            constants.REFERENCE_RESULTS[benchmark_name])
-        results[entry_name][benchmark_name] = result_val
-    self.report.set_results(results)
-
-  def get_compliance(self, filename):
-    """Get the compliance level of the output file."""
-    print('Running Compliance Check on {}'.format(filename))
-    print('#' * 80)
-    start_time, status, dt, qual, target = mlp_compliance.l2_check_file_w_starttime(
-        filename)
-    print('#' * 80)
-
-    if status:
-      level = '2'
-    else:
-      start_time, status, dt, qual, target = mlp_compliance.l1_check_file_w_starttime(
-          filename)
-      print('#' * 80)
-      if status:
-        level = '1'
-      else:
-        level = '0'
-
-    success = status and qual and target and qual >= target
-    return start_time, level, dt, qual, success
-
-  def verify_and_extract_time(self, log_file, division, result_name):
-    """Verifies and result and returns timing.
-
-    Uses submodule mlp_compliance (https://github.com/bitfort/mlp_compliance)
-
-    Args:
-      log_file: Absolute path to result file.
-      division: open, closed
-      result_name: name of the benchmark, ncf, ssd, etc
-
-    Returns:
-      Time for the result or `INFINITE_TIME` if not a success
-
-    Raises:
-      Exception: If expected compliance level is not hit or cannot figure
-      out expected compliance level.
-
-    """
-    expected_level = constants.DIVISION_COMPLIANCE_CHECK_LEVEL.get(
-        division, None)
-    print(result_name)
-    if expected_level is None:
-      raise Exception('Unknown division: {}'.format(division))
-    start_time, level, dt, _, success = self.get_compliance(log_file)
-    print(float(start_time))
-    if int(level) != expected_level:
-      raise Exception('Error Level {} does not match needed level {}:{}'.format(
-          level, expected_level, log_file))
-
-    # Sets failure to converge to "infinite time" per the rules
-    if success and dt:
-      return dt, start_time
-    else:
-      print('Result was not a success set to INFINITE_TIME({})'.format(
-          INFINITE_TIME))
-      return INFINITE_TIME, start_time
diff --git a/compliance/verify_submission/mlperf_submission_helper/constants.py b/compliance/verify_submission/mlperf_submission_helper/constants.py
deleted file mode 100644
index 4f79e6895..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/constants.py
+++ /dev/null
@@ -1,52 +0,0 @@
-"""Constants used throughput the library."""
-BENCHMARK_NAMES = [
-    "resnet", "ssd", "maskrcnn", "transformer", "gnmt", "ncf", "minigo"
-]
-SUBM_META_PROPS = ["org", "poc_email"]
-ENTRY_META_PROPS = [
-    "division", "status", "hardware", "framework", "interconnect", "nodes",
-    "os", "libraries", "compilers"
-    # Not required
-    # "power"
-    # "notes",
-]
-NODE_META_PROPS = [
-    "num_nodes", "cpu", "num_cores", "num_vcpus", "accelerator",
-    "num_accelerators", "sys_mem_size", "sys_storage_type", "sys_storage_size",
-    "cpu_accel_interconnect", "network_card", "num_network_cards", "notes"
-]
-REQUIRED_RESULT_NUM = {
-    "resnet": 5,
-    "ssd": 5,
-    "maskrcnn": 5,
-    "transformer": 10,
-    "gnmt": 10,
-    "ncf": 100,
-    "minigo": 20
-}
-
-REFERENCE_RESULTS = {
-    "resnet": 1.0,
-    "ssd": 1.0,
-    "maskrcnn": 1.0,
-    "transformer": 1.0,
-    "gnmt": 1.0,
-    "ncf": 1.0,
-    "minigo": 1.0
-}
-
-RESULT_SUBM_META_COLUMNS = [
-    "org",
-]
-RESULT_ENTRY_META_COLUMNS = [
-    "division", "status", "hardware", "framework",
-    # not required
-    # "power", "notes"
-]
-
-DIVISION_COMPLIANCE_CHECK_LEVEL = {"open": 1, "closed": 2}
-
-# check result status
-SUCCESS = "success"
-FAILURE = "failure"
-ERROR = "error"
diff --git a/compliance/verify_submission/mlperf_submission_helper/crypto.py b/compliance/verify_submission/mlperf_submission_helper/crypto.py
deleted file mode 100644
index c7aa3cdf3..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/crypto.py
+++ /dev/null
@@ -1,109 +0,0 @@
-import fnmatch
-import os
-import shutil
-
-from Cryptodome.PublicKey import RSA
-from Cryptodome.Random import get_random_bytes
-from Cryptodome.Cipher import AES, PKCS1_OAEP
-
-
-def encrypt_file(public_key, src_file, dest_file):
-  try:
-    with open(src_file) as f:
-      rsa_key = RSA.import_key(open(public_key).read())
-      session_key = get_random_bytes(16)
-      # Encrypt session key
-      cipher_rsa = PKCS1_OAEP.new(rsa_key)
-      encrypted_session_key = cipher_rsa.encrypt(session_key)
-      # Encrypt data
-      cipher_aes = AES.new(session_key, AES.MODE_EAX)
-      ciphertext, tag = cipher_aes.encrypt_and_digest(f.read().encode("utf-8"))
-  except Exception as e:
-    print("Unable to encrypt file: {}".format(src_file))
-    raise e
-
-  try:
-    with open(dest_file, "wb") as f:
-      for x in (encrypted_session_key, cipher_aes.nonce, tag, ciphertext):
-        f.write(x)
-  except Exception as e:
-    print("Unable to write output file {}".format(dest_file))
-    raise e
-
-
-def decrypt_file(private_key, src_file, dest_file):
-  try:
-    with open(src_file, "rb") as f:
-      rsa_key = RSA.import_key(open(private_key).read())
-      encrypted_session_key = f.read(rsa_key.size_in_bytes())
-      nonce = f.read(16)
-      tag = f.read(16)
-      ciphertext = f.read(-1)
-
-      # Decrypt session key
-      cipher_rsa = PKCS1_OAEP.new(rsa_key)
-      session_key = cipher_rsa.decrypt(encrypted_session_key)
-      # Decrypt data
-      cipher_aes = AES.new(session_key, AES.MODE_EAX, nonce)
-      data = cipher_aes.decrypt_and_verify(ciphertext, tag)
-      data = data.decode("utf-8")
-  except Exception as e:
-    print("Unable to decrypt file: {}".format(src_file))
-    raise e
-
-  try:
-    with open(dest_file, "w") as f:
-      f.write(data)
-  except Exception as e:
-    print("Unable to write output file: {}".format(dest_file))
-    raise e
-
-
-def encrypt_submission(key, src_dir, dest_dir):
-  if os.path.isdir(dest_dir):
-    raise Exception("Output directory already exists.")
-  os.mkdir(dest_dir, mode=0o755)
-  for root, dirs, files in os.walk(src_dir):
-    # identify result files and encrypt, else directly copy
-    if fnmatch.fnmatch(root, os.path.join(src_dir, "results", "*", "*")):
-      for f in files:
-        from_file = os.path.join(root, f)
-        to_file = from_file.replace(
-            src_dir.rstrip(os.sep), dest_dir.rstrip(os.sep), 1)
-        encrypt_file(key, from_file, to_file)
-    else:
-      for d in dirs:
-        from_dir = os.path.join(root, d)
-        to_dir = from_dir.replace(
-            src_dir.rstrip(os.sep), dest_dir.rstrip(os.sep), 1)
-        os.mkdir(to_dir, mode=0o755)
-      for f in files:
-        from_file = os.path.join(root, f)
-        to_file = from_file.replace(
-            src_dir.rstrip(os.sep), dest_dir.rstrip(os.sep), 1)
-        shutil.copyfile(from_file, to_file)
-
-
-def decrypt_submission(key, src_dir, dest_dir):
-  if os.path.isdir(dest_dir):
-    raise Exception("Output directory already exists.")
-  os.mkdir(dest_dir, mode=0o755)
-  for root, dirs, files in os.walk(src_dir):
-    # identify result files and encrypt, else directly copy
-    if fnmatch.fnmatch(root, os.path.join(src_dir, "results", "*", "*")):
-      for f in files:
-        from_file = os.path.join(root, f)
-        to_file = from_file.replace(
-            src_dir.rstrip(os.sep), dest_dir.rstrip(os.sep), 1)
-        decrypt_file(key, from_file, to_file)
-    else:
-      for d in dirs:
-        from_dir = os.path.join(root, d)
-        to_dir = from_dir.replace(
-            src_dir.rstrip(os.sep), dest_dir.rstrip(os.sep), 1)
-        os.mkdir(to_dir, mode=0o755)
-      for f in files:
-        from_file = os.path.join(root, f)
-        to_file = from_file.replace(
-            src_dir.rstrip(os.sep), dest_dir.rstrip(os.sep), 1)
-        shutil.copyfile(from_file, to_file)
diff --git a/compliance/verify_submission/mlperf_submission_helper/mlp_compliance b/compliance/verify_submission/mlperf_submission_helper/mlp_compliance
deleted file mode 160000
index 1178b3d45..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/mlp_compliance
+++ /dev/null
@@ -1,1 +0,0 @@
-Subproject commit 1178b3d45a5baa8cd9b15ac9771f09fef73e84a2
diff --git a/compliance/verify_submission/mlperf_submission_helper/report.py b/compliance/verify_submission/mlperf_submission_helper/report.py
deleted file mode 100644
index 78d441f06..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/report.py
+++ /dev/null
@@ -1,63 +0,0 @@
-import os
-
-from constants import *
-
-
-class SubmissionReport(object):
-  """SubmissionReport stores submission info and produces submission report."""
-
-  def __init__(self):
-    self.passed_checks = []
-    self.failed_checks = []
-    self.errors = []
-    self.results = {}
-
-  def add_passed_check(self, msg):
-    self.passed_checks.append(msg)
-
-  def add_failed_check(self, msg):
-    self.failed_checks.append(msg)
-
-  def add_error(self, msg):
-    self.errors.append(msg)
-
-  def set_results(self, results):
-    self.results = results
-
-  def print_report(self, verbose=True):
-
-    # print summary
-    print("\n" + "MLPERF SUBMISSION REPORT\n" + "========================\n")
-    print("\n" + "SUMMARY\n" + "-------\n")
-    print("Passed checks: {}\n".format(len(self.passed_checks)) +
-          "Failed checks: {}\n".format(len(self.failed_checks)) +
-          "Errors: {}".format(len(self.errors)))
-    print("Note: the Errors indicate certain checks being skipped or " +
-          "early terminated due to errors. The errors are likely caused " +
-          "by failed checks above.")
-
-    # print succeeded checks (in verbose mode only)
-    if verbose:
-      print("\n" + "PASSED CHECKS\n" + "-------------\n")
-      for msg in self.passed_checks:
-        print(msg)
-
-    # print failed checks
-    print("\n" + "FAILED CHECKS\n" + "-------------\n")
-    for msg in self.failed_checks:
-      print(msg)
-
-    # print errors
-    print("\n" + "ERRORS\n" + "------\n")
-    for msg in self.errors:
-      print(msg)
-
-  def print_results(self):
-    result_columns = RESULT_SUBM_META_COLUMNS + RESULT_ENTRY_META_COLUMNS + BENCHMARK_NAMES
-
-    print("\n" + "RESULTS\n" + "-------\n")
-    print(",".join(result_columns))
-    for entry_name in self.results:
-      value_list = [self.results[entry_name][key] for key in result_columns]
-      value_list = ["-" if val is None else str(val) for val in value_list]
-      print(",".join(value_list))
diff --git a/compliance/verify_submission/mlperf_submission_helper/utils.py b/compliance/verify_submission/mlperf_submission_helper/utils.py
deleted file mode 100644
index 53e88cf0b..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/utils.py
+++ /dev/null
@@ -1,14 +0,0 @@
-import os
-
-
-def get_subdirs(parent_path):
-  """Return a list of (name, path) tuples of direct subdirectories of
-
-    parent_path, where each tuple corresponds to one subdirectory. Files
-    in the parent_path are excluded from the output.
-    """
-  entries = os.listdir(parent_path)
-  subdirs = [(entry, os.path.join(parent_path, entry))
-             for entry in entries
-             if os.path.isdir(entry)]
-  return subdirs
diff --git a/compliance/verify_submission/mlperf_submission_helper/verify_submission.py b/compliance/verify_submission/mlperf_submission_helper/verify_submission.py
deleted file mode 100644
index 97d883bfc..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/verify_submission.py
+++ /dev/null
@@ -1,93 +0,0 @@
-import argparse
-import os
-import shutil
-import sys
-
-import checks as submission_checks
-import constants
-import report
-
-
-def verify_submission(args):
-  root_dir = args.root
-  public_key = args.public_key
-  private_key = args.private_key
-  encrypt_out = args.encrypt_out
-  decrypt_out = args.decrypt_out
-
-  # validate args
-  if any([public_key, encrypt_out]) and not all([public_key, encrypt_out]):
-    print("--encrypt-key and --encrypt-out must be present togetger.")
-    sys.exit(1)
-  if any([private_key, decrypt_out]) and not all([private_key, decrypt_out]):
-    print("--decrypt-key and --decrypt-out must be present together.")
-    sys.exit(1)
-  if all([private_key, public_key]):
-    print("--encrypt-key and --decrypt-key cannot be present together.")
-    sys.exit(1)
-
-  if any([public_key, private_key]):
-    import crypto
-
-  # if decrypt-key is provided, then decrypt the submission, save it to
-  # decrypt-out and point submission root to the decrypted directory
-  if private_key:
-    try:
-      crypto.decrypt_submission(private_key, root_dir, decrypt_out)
-    except Exception as e:
-      print("Unable to decrypt submission: {}".format(str(e)))
-      sys.exit(1)
-    print("Decrypted submission saved at {}".format(decrypt_out))
-    root_dir = decrypt_out
-
-  # perform verifications and extract results
-  checks = submission_checks.SubmissionChecks()
-  checks.verify_dirs_and_files(root_dir)
-  checks.verify_metadata()
-  checks.compile_results()
-
-  checks.report.print_report()
-  checks.report.print_results()
-
-  # if encrypt-key is provided, then encrypt the submission
-  # and save it to encrypt-out
-  if public_key:
-    try:
-      crypto.encrypt_submission(public_key, root_dir, encrypt_out)
-    except Exception as e:
-      print("Unable to encrypt submission: {}".format(str(e)))
-      sys.exit(1)
-    print("Encrypted submission saved at {}".format(encrypt_out))
-
-
-def main():
-  parser = argparse.ArgumentParser(description="Verify MLPerf submission.")
-  parser.add_argument(
-      "root", metavar="SUBMISSION_ROOT", help="submission root directory")
-  parser.add_argument(
-      "--encrypt-key",
-      dest="public_key",
-      default=None,
-      help="public key for encrypting log files")
-  parser.add_argument(
-      "--encrypt-out",
-      dest="encrypt_out",
-      default=None,
-      help="output path for encrypted submission")
-  parser.add_argument(
-      "--decrypt-key",
-      dest="private_key",
-      default=None,
-      help="private key for decrypting log files")
-  parser.add_argument(
-      "--decrypt-out",
-      dest="decrypt_out",
-      default=None,
-      help="output path for decrypted submission")
-  args = parser.parse_args()
-
-  verify_submission(args)
-
-
-if __name__ == "__main__":
-  main()
diff --git a/data_generation/fractal_graph_expansions/README.md b/data_generation/fractal_graph_expansions/README.md
deleted file mode 100644
index 86a38443c..000000000
--- a/data_generation/fractal_graph_expansions/README.md
+++ /dev/null
@@ -1,119 +0,0 @@
-Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-     http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-
-
-
-This is a directory used to produce larger versions of MovieLens through
-  fractal expansions. The expansion is stochastic, see equation (2) and
-  algorithm (1) in https://arxiv.org/abs/1901.08910 for more details
-  (temporary, to be replaced by white paper). The ratings are binarized,
-  each rating becoming 1.
-
-If you use this data please cite:
-  * https://dl.acm.org/citation.cfm?doid=2866565.2827872
-  * https://arxiv.org/abs/1901.08910 (temporary, to be replaced by white paper)
-
-Dependencies:
-  * numpy (tested with version 1.16.1)
-  * absl-py (tested with version 0.7.0)
-  * pandas (tested with version 0.24.1)
-  * scipy (tested with version 1.2.1)
-  * tensorflow (tested with version 1.12.0)
-  * scikit-image (tested with version 0.14.2)
-  * sklearn (tested with version 0.20.2)
-
-How to run (takes ~30 mins on a recent desktop):
-  1) Download MovieLens20m from https://grouplens.org/datasets/movielens/
-  (permalink http://grouplens.org/datasets/movielens/20m/).
-  2) python run_expansion.py will generate the data set with the following flags
-  * --input_csv_file, the path to the ratings.csv file downloaded
-    from MovieLens.
-  * --num_row_multiplier, the multiplier for the number of users.
-    16 (default) yields ~1B interactions, for now 4 is used to train models.
-  * --num_col_multiplier, the multiplier for the number of items.
-    32 (default) yields ~1B interactions, for now 16 is used to train models.
-  * --output_prefix, the path to the output files including their prefix.
-
-Sizes of generated data sets:
-  1) With --num_row_multiplier=16 --num_col_multiplier=32:
-  * 1,223,962,043 interactions in train set
-  * 12,709,557 interactions in test set
-  * 2,197,225 users
-  * 855,776 items
-  2) With --num_row_multiplier=4 --num_col_multiplier=16:
-  * 131,203,749 interactions in train set
-  * 1,462,391 interactions in test set
-  * 498,975 users
-  * 427,888 items
-
-A train and test set will be generated. No information from the test set is
-  available in the train set.
-The train set data will consist of num_row_multiplier shards named
-  output_prefix_train.pkl_%d % shard for shard in range(num_row_multiplier).
-The test set data will consist of num_row_multiplier shards named
-  output_prefix_test.pkl_%d % shard for shard in range(num_row_multiplier).
-Each shard is a pickled list of numpy arrays, each array corresponds to an user
-  and entails the sequence of item indices corresponding to the items the user
-  has interacted with.
-The train and test sets will also each feature a separate metadata file
-  output_prefix_train/test_metadata.pkl. The metadata contains a pickled
-  graph_expansion.SparseMatrixMetadata object entailing the number of
-  interactions, users and items in each data set. (Don't forget to import
-  graph_expansion.SparseMatrixMetadata before pickle.load(...)).
-
-If the original rating matrix (after filtering) has (n, m) (users, items) then
-  the synthesized matrix will have about
-  (n x num_row_multiplier, m x num_col_multiplier) (users, items).
-
-Actual users with less than two distinct rating timestamps are dropped from
-  the original data set. Synthetic users with no ratings in either the synthetic
-  train or test set are all dropped. Items without ratings may be present in
-  the train and/or test sets.
-
-Other useful flags:
-  1) --min_dropout_rate, decreasing/increasing this value will result in
-    a denser/sparser generated data set. 0.05 (default) is used.
-  2) --max_dropout_rate, decreasing/increasing this value will result in
-    a denser/sparser generated data set. 0.99 (default) is used.
-
-# Running instructions for the recommendation benchmark
-
-### Steps to download and verify data
-
-You can download and verify the dataset by running the `download_dataset.sh` and `verify_dataset.sh` scripts from the parent `recommendation` directory.
-Assume you want to store the downloaded dataset in `/my_data_dir` directory:
-
-1. Install `unzip` and `curl`.
-2. Download and unzip `ml-20m.zip`:
-```bash
-mkdir /my_data_dir
-cd /my_data_dir
-# Creates ml-20.zip
-source <PATH_TO_RECOMMENDATION_DIR>/download_dataset.sh
-# Confirms the MD5 checksum of ml-20.zip
-source <PATH_TO_RECOMMENDATION_DIR>/verify_dataset.sh
-unzip ml-20m.zip
-```
-
-### Step to expand the dataset (x16 users, x32 items)
-
-Assuming that the unzipped ML-20M dataset is stored under `/my_data_dir/ml-20m`, 
-go to `data_generation/fractal_graph_expansions` directory and run:
-
-```bash
-pip install -r requirements.txt
-DATA_DIR=/my_data_dir ./data_gen.sh
-```
-
-The resulting dataset should be stored under `/my_data_dir/ml-20mx16x32`.
diff --git a/data_generation/fractal_graph_expansions/data_gen.sh b/data_generation/fractal_graph_expansions/data_gen.sh
deleted file mode 100755
index 877c94c18..000000000
--- a/data_generation/fractal_graph_expansions/data_gen.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-DATASET=${DATASET:-ml-20m}
-USER_MUL=${USER_MUL:-16}
-ITEM_MUL=${ITEM_MUL:-32}
-DATA_DIR=${DATA_DIR:-/data/cache}
-
-DATA_PATH=${DATA_DIR}/${DATASET}x${USER_MUL}x${ITEM_MUL}/
-
-mkdir -p ${DATA_PATH}
-python run_expansion.py --input_csv_file ${DATA_DIR}/${DATASET}/ratings.csv --num_row_multiplier ${USER_MUL} --num_col_multiplier ${ITEM_MUL} --output_prefix ${DATA_PATH}
diff --git a/data_generation/fractal_graph_expansions/graph_analysis.py b/data_generation/fractal_graph_expansions/graph_analysis.py
deleted file mode 100644
index dc5a8ef8f..000000000
--- a/data_generation/fractal_graph_expansions/graph_analysis.py
+++ /dev/null
@@ -1,65 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Toolbox for graph adjancency/incidence matrix analysis.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from absl import flags
-from scipy.sparse import linalg
-
-flags.DEFINE_integer("max_iter_sparse_svd",
-                     16,
-                     "Maximum number of iterations in sparse "
-                     "Singular Value Decomposition through power iterations.")
-
-FLAGS = flags.FLAGS
-
-
-def sparse_svd(sparse_matrix, num_values, max_iter):
-  """Wrapper around SciPy's Singular Value Decomposition for sparse matrices.
-
-  Args:
-    sparse_matrix: a SciPy sparse matrix (typically large).
-    num_values: the number of largest singular values to compute.
-    max_iter: maximum number of iterations (>= 0) in the decomposition. If
-      max_iter is None, runs FLAGS.max_iter_sparse_svd steps. If max_iter == 0,
-      runs until convergence. Otherwise will run max_iter steps.
-
-  Returns:
-    A (u, s, v) tuple where s is an array entailing the singular values,
-      and (u, v) the singular vector matrices. u is column orthogonal and
-      v is row orthogonal. s is sorted in increasing order.
-  """
-
-  if num_values <= 0:
-    raise ValueError("num_values should be > 0 but instead is %d." % num_values)
-
-  if max_iter is not None and max_iter < 0:
-    raise ValueError("max_iter should be >= 0 but instead is %d." % max_iter)
-
-  if max_iter is None:
-    max_iter = FLAGS.max_iter_sparse_svd
-  elif not max_iter:
-    max_iter = None
-
-  u, s, v = linalg.svds(
-      sparse_matrix, k=num_values, maxiter=max_iter,
-      return_singular_vectors=True)
-
-  return (u, s, v)
-
diff --git a/data_generation/fractal_graph_expansions/graph_expansion.py b/data_generation/fractal_graph_expansions/graph_expansion.py
deleted file mode 100644
index 90492a033..000000000
--- a/data_generation/fractal_graph_expansions/graph_expansion.py
+++ /dev/null
@@ -1,303 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Fractal graph expander.
-
-Detailed analysis in the deterministic case provided in
-https://arxiv.org/abs/1901.08910.
-Please refer the paper if you use this code.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import collections
-
-from six.moves import xrange
-
-from absl import logging
-
-from scipy import sparse
-
-import util
-from random_matrix_ops import shuffle_sparse_coo_matrix
-
-
-SparseMatrixMetadata = collections.namedtuple(
-    "SparseMatrixMetadata", [
-        "num_interactions",
-        "num_rows",
-        "num_cols",
-    ])
-
-
-def _compute_and_write_row_block(
-    i, left_matrix, right_matrix, train_indices_out_path, test_indices_out_path,
-    remove_empty_rows):
-  """Compute row block (shard) of expansion for row i of the left_matrix.
-
-  Compute a shard of the randomized Kronecker product and dump it on the fly.
-  A standard Kronecker product between matrices A and B produces
-                        [[a_11 B, ..., a_1n B],
-                                  ...
-                         [a_m1 B, ..., a_mn B]]
-    (if A's size is (m, n) and B's size is (p, q) then A Kronecker B has size
-    (m p, n q)).
-    Here we modify the standard Kronecker product expanding matrices in
-    https://cs.stanford.edu/~jure/pubs/kronecker-jmlr10.pdf
-    and randomize each block-wise operation a_ij B in the Kronecker product as
-    in https://arxiv.org/pdf/1901.08910.pdf section III.4.
-    The matrix we produce is
-                       [[F(a_11, B, w_11), ..., F(a_1n, B, w_1n)],
-                                           ...
-                       [F(a_m1, B, w_m1), ... , F(a_mn, B, w_mn)]]
-    where (w_ij) is a sequence of pseudo random numbers and F is randomized
-    operator which will:
-      1) Shuffle rows and columns of B independently at random;
-      2) Dropout elements of B with a rate 1 - a_ij to compute
-        F(a_ij, B, w_ij).
-    (It is noteworthy that there is an abuse of notation above when writing
-    F(a_ij, B, w_ij) as each block-wise operation will in fact consume
-    multiple elements of the sequence (w_ij)).
-  Each shard of index i consists of [F(a_i1, B, w_i1), ..., F(a_in, B, w_in)]
-
-  Args:
-    i: index of the shard. The rows i * m to (i + 1) * m of the full synthetic
-      matrix matrix will be computed and dumpted to file.
-    left_matrix: sparse SciPy csr matrix with values in [0, 1].
-    right_matrix: sparse SciPy coo signed binary matrix. +1 values correspond
-      to train set and -1 values correspond to test set.
-    train_indices_out_path: path to output train file. The non zero indices of
-      the resulting sparse matrix are dumped as a series of pickled records.
-      '_i' will be used as a suffix for the shard's output file. The shard
-      contains a pickled list of list each of which corresponds to a users.
-    test_indices_out_path: path to output train file. The non zero indices of
-      the resulting sparse matrix are dumped as a series of pickled records.
-     '_i' will be used as a suffix for the shard's output file. The shard
-      contains a pickled list of list each of which corresponds to a users.
-    remove_empty_rows: whether to remove rows from the synthetic train and
-      test matrices which are not present in the train or the test matrix.
-
-  Returns:
-    (num_removed_rows, metadata, train_metadata, test_metadata): an integer
-      specifying the number of rows dropped because of dropout followed by
-      a triplet of SparseMatrixMetadata corresponding to the overall shard,
-      train shard and test shard.
-  """
-
-  kron_blocks = []
-
-  num_rows = 0
-  num_removed_rows = 0
-  num_interactions = 0
-  num_train_interactions = 0
-  num_test_interactions = 0
-
-  # Construct blocks
-  for j in xrange(left_matrix.shape[1]):
-
-    dropout_rate = 1.0 - left_matrix[i, j]
-    kron_block = shuffle_sparse_coo_matrix(right_matrix, dropout_rate)
-
-    if not set(kron_block.data).issubset({1, -1}):
-      raise ValueError("Values of sparse matrix should be -1 or 1 but are: ",
-                       set(kron_block.data))
-
-    kron_blocks.append(kron_block)
-
-    logging.info("Done with element (%d, %d)", i, j)
-
-  rows_to_write = sparse.hstack(kron_blocks).tocoo()
-
-  train_rows_to_write = util.sparse_where_equal(rows_to_write, 1)
-  test_rows_to_write = util.sparse_where_equal(rows_to_write, -1)
-
-  logging.info("Producing data set row by row")
-
-  all_train_items_to_write = []
-  all_test_items_to_write = []
-  # Write Kronecker product line per line.
-  for k in xrange(right_matrix.shape[0]):
-
-    train_items_to_write = train_rows_to_write.getrow(k).indices
-    test_items_to_write = test_rows_to_write.getrow(k).indices
-
-    # for users with > 1 test items, keep only the first one
-    if len(test_items_to_write) > 1:
-        test_items_to_write = test_items_to_write[:1]
-
-    num_train = train_items_to_write.shape[0]
-    num_test = test_items_to_write.shape[0]
-
-    if remove_empty_rows and ((not num_train) or (not num_test)):
-      logging.info("Removed empty output row %d.",
-                   i * left_matrix.shape[0] + k)
-      num_removed_rows += 1
-      continue
-
-    num_rows += 1
-    num_interactions += num_train + num_test
-    num_train_interactions += num_train
-    num_test_interactions += num_test
-
-    all_train_items_to_write.append(train_items_to_write)
-    all_test_items_to_write.append(test_items_to_write)
-
-    if k % 1000 == 0:
-      logging.info("Done producing data set row %d.", k)
-
-  logging.info("Done producing data set row by row.")
-
-  util.savez_two_column(
-      all_train_items_to_write, 
-      row_offset=(i * right_matrix.shape[0]),
-      file_name=train_indices_out_path + ("_%d" % i))
-  util.savez_two_column(
-      all_test_items_to_write, 
-      row_offset=(i * right_matrix.shape[0]),
-      file_name=test_indices_out_path + ("_%d" % i))
-
-  num_cols = rows_to_write.shape[1]
-  metadata = SparseMatrixMetadata(num_interactions=num_interactions,
-                                  num_rows=num_rows, num_cols=num_cols)
-  train_metadata = SparseMatrixMetadata(num_interactions=num_train_interactions,
-                                        num_rows=num_rows, num_cols=num_cols)
-  test_metadata = SparseMatrixMetadata(num_interactions=num_test_interactions,
-                                       num_rows=num_rows, num_cols=num_cols)
-
-  logging.info("Done with left matrix row %d.", i)
-  logging.info("%d interactions written in shard.", num_interactions)
-  logging.info("%d rows removed in shard.", num_removed_rows)
-  logging.info("%d train interactions written in shard.",
-               num_train_interactions)
-  logging.info("%d test interactions written in shard.",
-               num_test_interactions)
-
-  return (num_removed_rows, metadata, train_metadata, test_metadata)
-
-
-def output_randomized_kronecker_to_pickle(
-    left_matrix, right_matrix,
-    train_indices_out_path, test_indices_out_path,
-    train_metadata_out_path=None, test_metadata_out_path=None,
-    remove_empty_rows=True):
-  """Compute randomized Kronecker product and dump it on the fly.
-
-  A standard Kronecker product between matrices A and B produces
-                        [[a_11 B, ..., a_1n B],
-                                  ...
-                         [a_m1 B, ..., a_mn B]]
-    (if A's size is (m, n) and B's size is (p, q) then A Kronecker B has size
-    (m p, n q)).
-    Here we modify the standard Kronecker product expanding matrices in
-    https://cs.stanford.edu/~jure/pubs/kronecker-jmlr10.pdf
-    and randomize each block-wise operation a_ij B in the Kronecker product as
-    in https://arxiv.org/pdf/1901.08910.pdf section III.4.
-    The matrix we produce is
-                       [[F(a_11, B, w_11), ..., F(a_1n, B, w_1n)],
-                                           ...
-                       [F(a_m1, B, w_m1), ... , F(a_mn, B, w_mn)]]
-    where (w_ij) is a sequence of pseudo random numbers and F is randomized
-    operator which will:
-      1) Shuffle rows and columns of B independently at random;
-      2) Dropout elements of B with a rate 1 - a_ij to compute
-        F(a_ij, B, w_ij).
-    (It is noteworthy that there is an abuse of notation above when writing
-    F(a_ij, B, w_ij) as each block-wise operation will in fact consume
-    multiple elements of the sequence (w_ij)).
-
-  Args:
-    left_matrix: sparse SciPy csr matrix with values in [0, 1].
-    right_matrix: sparse SciPy coo signed binary matrix. +1 values correspond
-      to train set and -1 values correspond to test set.
-    train_indices_out_path: path to output train file. The non zero indices of
-      the resulting sparse matrix are dumped as a series of pickled records.
-      As many shard will be created as there are rows in left matrix. The shard
-      corresponding to row i in the left matrix has the suffix _i appended to
-      its file name. Each shard contains a pickled list of list each of which
-      corresponds to a users.
-    test_indices_out_path: path to output train file. The non zero indices of
-      the resulting sparse matrix are dumped as a series of pickled records.
-      As many shard will be created as there are rows in left matrix. The shard
-      corresponding to row i in the left matrix has the suffix _i appended to
-      its file name. Each shard contains a pickled list of list each of which
-      corresponds to a users.
-    train_metadata_out_path: path to optional complementary output file
-      containing the number of train rows (r), columns (c) and non zeros (nnz)
-      in a pickled SparseMatrixMetadata named tuple.
-    test_metadata_out_path: path to optional complementary output file
-      containing the number of test rows (r), columns (c) and non zeros (nnz)
-      in a pickled SparseMatrixMetadata named tuple.
-    remove_empty_rows: whether to remove rows from the synthetic train and
-      test matrices which are not present in the train or the test matrix.
-
-  Returns:
-    (metadata, train_metadata, test_metadata) triplet of SparseMatrixMetadata
-      corresponding to the overall data set, train data set and test data set.
-  """
-  logging.info("Writing item sequences to pickle files %s and %s.",
-               train_indices_out_path, test_indices_out_path)
-
-  num_rows = 0
-  num_removed_rows = 0
-  num_cols = left_matrix.shape[1] * right_matrix.shape[1]
-  num_interactions = 0
-
-  num_train_interactions = 0
-  num_test_interactions = 0
-
-  if not set(right_matrix.data).issubset({-1, 1}):
-    raise ValueError(
-        "Values of sparse matrix should be -1 or 1 but are:",
-        set(right_matrix.data))
-
-  for i in xrange(left_matrix.shape[0]):
-
-    (shard_num_removed_rows, shard_metadata, shard_train_metadata,
-     shard_test_metadata) = _compute_and_write_row_block(
-         i, left_matrix, right_matrix, train_indices_out_path,
-         test_indices_out_path, remove_empty_rows)
-
-    num_rows += shard_metadata.num_rows
-    num_removed_rows += shard_num_removed_rows
-    num_interactions += shard_metadata.num_interactions
-    num_train_interactions += shard_train_metadata.num_interactions
-    num_test_interactions += shard_test_metadata.num_interactions
-
-    logging.info("%d total interactions written.", num_interactions)
-    logging.info("%d total rows removed.", num_removed_rows)
-    logging.info("%d total train interactions written.", num_train_interactions)
-    logging.info("%d toal test interactions written.", num_test_interactions)
-
-  logging.info("Done writing.")
-
-  metadata = SparseMatrixMetadata(
-      num_interactions=num_interactions,
-      num_rows=num_rows, num_cols=num_cols)
-  train_metadata = SparseMatrixMetadata(
-      num_interactions=num_train_interactions,
-      num_rows=num_rows, num_cols=num_cols)
-  test_metadata = SparseMatrixMetadata(
-      num_interactions=num_test_interactions,
-      num_rows=num_rows, num_cols=num_cols)
-
-  if train_metadata_out_path is not None:
-    util.write_metadata_to_file(
-        train_metadata, train_metadata_out_path, tag="train")
-  if test_metadata_out_path is not None:
-    util.write_metadata_to_file(
-        test_metadata, test_metadata_out_path, tag="test")
-
-  return metadata, train_metadata, test_metadata
diff --git a/data_generation/fractal_graph_expansions/graph_reduction.py b/data_generation/fractal_graph_expansions/graph_reduction.py
deleted file mode 100644
index c0b35f1b1..000000000
--- a/data_generation/fractal_graph_expansions/graph_reduction.py
+++ /dev/null
@@ -1,64 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Toolbox to reduce a large matrix into a smaller yet similar matrix.
-
-Given a large matrix R of size (m, n), the toolbox produces a matrix R_hat which
-is much smaller but is similar in the largest singular values.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import numpy as np
-import scipy.linalg
-
-import skimage.transform as transform
-
-
-def _closest_column_orthogonal_matrix(matrix):
-  return np.matmul(matrix, np.linalg.inv(
-      scipy.linalg.sqrtm(np.matmul(matrix.T, matrix))))
-
-
-def resize_matrix(usv, num_rows, num_cols):
-  """Apply algorith 2 in https://arxiv.org/pdf/1901.08910.pdf.
-
-  Args:
-    usv: matrix to reduce given in SVD form with the spectrum s in
-      increasing order.
-    num_rows: number of rows in the output matrix.
-    num_cols: number of columns in the output matrix.
-  Returns:
-    A resized version of (u, s, v) whose non zero singular values will be
-      identical to the largest singular values in s.
-  """
-  u, s, v = usv
-  k = min(num_rows, num_cols)
-
-  u_random_proj = transform.resize(u[:, :k], (num_rows, k))
-  v_random_proj = transform.resize(v[:k, :], (k, num_cols))
-
-  u_random_proj_orth = _closest_column_orthogonal_matrix(u_random_proj)
-  v_random_proj_orth = _closest_column_orthogonal_matrix(v_random_proj.T).T
-
-  return np.matmul(u_random_proj_orth,
-                   np.matmul(np.diag(s[::-1][:k]), v_random_proj_orth))
-
-
-def normalize_matrix(matrix):
-  """Fold all values of the matrix into [0, 1]."""
-  abs_matrix = np.abs(matrix.copy())
-  return abs_matrix / abs_matrix.max()
diff --git a/data_generation/fractal_graph_expansions/post_process.py b/data_generation/fractal_graph_expansions/post_process.py
deleted file mode 100644
index f38eae382..000000000
--- a/data_generation/fractal_graph_expansions/post_process.py
+++ /dev/null
@@ -1,67 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Post process expanded data set to output to user/item/rating csv files."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from absl import app
-from absl import flags
-from absl import logging
-
-import pickle
-import pandas as pd
-import tensorflow as tf
-
-
-flags.DEFINE_string("output_prefix",
-                    "",
-                    "Prefix to the path of the pickle files that have been "
-                    "produced. output_prefix_train.csv and "
-                    "output_prefix_test.csv will be created.")
-flags.DEFINE_integer("num_shards",
-                     16,
-                     "Number of shards used to output data.")
-
-FLAGS = flags.FLAGS
-
-
-def _read_from_serialized_file(file_name):
-  with tf.gfile.Open(file_name, "rb") as infile:
-    return pickle.load(infile)
-
-
-def _convert_pickled_shards_to_csv(prefix):
-  logging.info("Processing %d shards with prefix %s.", FLAGS.num_shards, prefix)
-  for shard_idx in range(FLAGS.num_shards):
-    shard = _read_from_serialized_file(prefix + ".pkl_%d" % shard_idx)
-    logging.info("Converting shard %d to csv format.", shard_idx)
-    with tf.gfile.Open(prefix + ".csv_%d" % shard_idx, "wb") as outfile:
-      for user_idx, item_array in enumerate(shard):
-        for item_idx in item_array:
-          outfile.write("%d,%d,1\n" % (user_idx, item_idx))
-    logging.info("Done converting shard %d to csv format.", shard_idx)
-  logging.info("Done processing %d shards with prefix %s.", FLAGS.num_shards, prefix)
-
-
-def main(_):
-  
-   _convert_pickled_shards_to_csv(FLAGS.output_prefix + "_train")
-   _convert_pickled_shards_to_csv(FLAGS.output_prefix + "_test")
-
-
-if __name__ == "__main__":
-  app.run(main)
diff --git a/data_generation/fractal_graph_expansions/random_matrix_ops.py b/data_generation/fractal_graph_expansions/random_matrix_ops.py
deleted file mode 100644
index 913d35a82..000000000
--- a/data_generation/fractal_graph_expansions/random_matrix_ops.py
+++ /dev/null
@@ -1,139 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Fast shuffle and dropout of large sparse matrices.
-
-For speed matrices are encoded in a pandas dataframe. To make the behavior
-of the following operators deterministic, it is sufficient to setup numpy's
-random seed before these operators are called (numpy.random.seed(seed_value)).
-Note also that callers running the functions below in parallel are responsible
-for guaranteeing that the corresponding underlying sequences of random numbers
-(which will be genereted in parallel) are non overlapping.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-
-from absl import flags
-from absl import logging
-
-import numpy as np
-
-from scipy import sparse
-from sklearn.utils import shuffle
-
-flags.DEFINE_float("min_dropout_rate",
-                   0.05,
-                   "Mininum dropout rate in shuffle_sparse_matrix if none is "
-                   "specified. A lower dropout rate will be clipped to "
-                   "the minimum value.")
-flags.DEFINE_float("max_dropout_rate",
-                   0.99,
-                   "Maximum dropout rate in shuffle_sparse_matrix if none is "
-                   "specified. A greater dropout rate will be clipped to "
-                   "the maximum value.")
-
-FLAGS = flags.FLAGS
-
-
-def _dropout_sparse_coo_matrix(sparse_matrix, rate,
-                               min_dropout_rate, max_dropout_rate):
-  """Drop values from a sparse matrix encoded as a SciPy coo matrix.
-
-  Args:
-    sparse_matrix: a SciPy coo sparse matrix.
-    rate: if rate > 0 then non-zero elements of the input matrix
-      will be droped uniformly at random.
-    min_dropout_rate: minimum value for the dropout rate. If None
-      FLAGS.min_dropout_rate is used. If dropout_rate is lower than
-      min_dropout_rate it will clipped to min_dropout_rate.
-    max_dropout_rate: minimum value for the dropout rate. If None
-      FLAGS.max_dropout_rate is used. If dropout_rate is greater than
-      max_dropout_rate it will clipped to max_dropout_rate.
-
-  Returns:
-    A SciPy coo matrix containing those non zero elements that have not been
-    dropped out.
-  """
-  if min_dropout_rate is None:
-    min_dropout_rate = FLAGS.min_dropout_rate
-
-  if max_dropout_rate is None:
-    max_dropout_rate = FLAGS.max_dropout_rate
-
-  if min_dropout_rate > max_dropout_rate:
-    raise ValueError("min_dropout_rate (%f) should be less or equal to "
-                     "max_dropout_rate (%f)"
-                     % (min_dropout_rate, max_dropout_rate))
-
-  max_frac = 1.0 - min_dropout_rate
-  min_frac = 1.0 - max_dropout_rate
-  sampling_rate = 1.0 - rate
-
-  sampled_fraction = min(max(sampling_rate, min_frac), max_frac)
-  if sampled_fraction != sampling_rate:
-    logging.warning("Minimum sampling rate is %2f.", min_frac)
-    logging.warning("Maximum sampling rate is %2f.", max_frac)
-    logging.warning("Desired sampling rate is %2f.", sampling_rate)
-    logging.warning("Desired sampling rate %2f clipped to %2f.", sampling_rate,
-                    sampled_fraction)
-
-  num_sampled = min(
-      max(int(sparse_matrix.nnz * sampled_fraction), 1), sparse_matrix.nnz)
-  sampled_indices = np.random.choice(sparse_matrix.nnz, size=num_sampled,
-                                     replace=False)
-
-  return sparse.coo_matrix((sparse_matrix.data[sampled_indices],
-                            (sparse_matrix.row[sampled_indices],
-                             sparse_matrix.col[sampled_indices])),
-                           shape=sparse_matrix.shape)
-
-
-def shuffle_sparse_coo_matrix(sparse_matrix, dropout_rate=0.0,
-                              min_dropout_rate=None, max_dropout_rate=None):
-  """Shuffle sparse matrix encoded as a SciPy coo matrix.
-
-  Args:
-    sparse_matrix: a SciPy coo sparse matrix.
-    dropout_rate: if dropout_rate > 0 then non-zero elements of the input matrix
-      will be droped uniformly at random.
-    min_dropout_rate: minimum value for the dropout rate. If None
-      FLAGS.min_dropout_rate is used.
-    max_dropout_rate: minimum value for the dropout rate. If None
-      FLAGS.max_dropout_rate is used.
-
-  Returns:
-    A SciPy csr_matrix entailing the randomized interactions.
-  """
-
-  if (dropout_rate < 0.0) or (dropout_rate >= 1.0):
-    raise ValueError("Dropout rate should be in [0, 1) but is %f"
-                     % dropout_rate)
-
-  (num_rows, num_cols) = sparse_matrix.shape
-  shuffled_rows = shuffle(np.arange(num_rows))
-  shuffled_cols = shuffle(np.arange(num_cols))
-
-  if dropout_rate > 0.0:
-    sparse_matrix = _dropout_sparse_coo_matrix(
-        sparse_matrix, dropout_rate, min_dropout_rate, max_dropout_rate)
-
-  new_row = np.take(shuffled_rows, sparse_matrix.row)
-  new_col = np.take(shuffled_cols, sparse_matrix.col)
-
-  return sparse.csr_matrix(
-      (sparse_matrix.data, (new_row, new_col)), shape=(num_rows, num_cols))
diff --git a/data_generation/fractal_graph_expansions/requirements.txt b/data_generation/fractal_graph_expansions/requirements.txt
deleted file mode 100644
index 0976e9c47..000000000
--- a/data_generation/fractal_graph_expansions/requirements.txt
+++ /dev/null
@@ -1,8 +0,0 @@
-absl-py==0.7.0
-numpy==1.16.2
-pandas==0.24.2
-scikit-image=0.14.2
-scikit-learn=0.20.3
-scipy==1.2.1
-six==1.12.0
-tensorflow==1.13.1
diff --git a/data_generation/fractal_graph_expansions/run.sh b/data_generation/fractal_graph_expansions/run.sh
deleted file mode 100644
index a2a0272bb..000000000
--- a/data_generation/fractal_graph_expansions/run.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-#!/bin/bash
-
-input_file="/data/ml-20m/ratings.csv"
-output_prefix=""
-num_row_multiplier=16
-num_col_multiplier=32
-
-python run_expansion.py --input_csv_file=$input_file --output_prefix=$output_prefix --num_row_multiplier=$num_row_multiplier --num_col_multiplier=$num_col_multiplier
-# python post_process.py --output_prefix=$output_prefix --num_shards=$num_row_multiplier
diff --git a/data_generation/fractal_graph_expansions/run_expansion.py b/data_generation/fractal_graph_expansions/run_expansion.py
deleted file mode 100644
index c2df4da74..000000000
--- a/data_generation/fractal_graph_expansions/run_expansion.py
+++ /dev/null
@@ -1,202 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Run fractal expansion of MovieLens 20m.
-
-Detailed analysis of the deterministic case provided in
-https://arxiv.org/abs/1901.08910.
-Please refer the paper if you use this code.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-
-from absl import app
-from absl import flags
-from absl import logging
-
-import numpy as np
-
-import util
-from graph_analysis import sparse_svd
-from graph_expansion import output_randomized_kronecker_to_pickle
-from graph_reduction import normalize_matrix
-from graph_reduction import resize_matrix
-
-
-flags.DEFINE_string("input_csv_file",
-                    "ratings.csv",
-                    "Rating file of MovieLens 20m.")
-flags.DEFINE_integer("num_row_multiplier",
-                     16,
-                     "Factor by which the number of rows in the rating "
-                     "matrix will be multiplied.")
-flags.DEFINE_integer("num_col_multiplier",
-                     32,
-                     "Factor by which the number of columns in the rating "
-                     "matrix will be multiplied.")
-flags.DEFINE_string("output_prefix",
-                    "",
-                    "Prefix to the path of the files that will be "
-                    "produced. output_prefix/trainxAxB_C.npz and "
-                    "output_prefix/testxAxB_C.npz will be created, "
-                    "where A is num_row_multiplier, B is num_col_multiplier, "
-                    "and C goes from 0 to (num_row_multiplier - 1).")
-flags.DEFINE_integer("random_seed",
-                     0,
-                     "Random seed for all random operations.")
-
-FLAGS = flags.FLAGS
-
-
-def _create_index(df, colname):
-  value_set = sorted(set(df[colname].values))
-  num_unique = len(value_set)
-  return dict(zip(value_set, range(num_unique)))
-
-
-def _create_row_col_indices(ratings_df):
-  """Maps user and items ids to their locations in the rating matrix."""
-  user_id_to_user_idx = _create_index(ratings_df, "userId")
-  item_id_to_item_idx = _create_index(ratings_df, "movieId")
-
-  ratings_df["row"] = ratings_df["userId"].apply(
-      lambda x: user_id_to_user_idx[x])
-  ratings_df["col"] = ratings_df["movieId"].apply(
-      lambda x: item_id_to_item_idx[x])
-
-  return ratings_df
-
-
-def _preprocess_movie_lens(ratings_df):
-  """Separate the rating datafram into train and test sets.
-
-  Filters out users with less than two distinct timestamps. Creates train set
-  and test set. The test set contains all the last interactions of users with
-  more than two distinct timestamps.
-
-  Args:
-    ratings_df: pandas dataframe with columns 'userId', 'movieId', 'rating',
-      'timestamp'.
-
-  Returns:
-    tuple of dataframes (filtered_ratings, train_ratings, test_ratings).
-  """
-  ratings_df["data"] = 1.0
-  num_timestamps = ratings_df[["userId", "timestamp"]].groupby(
-      "userId").nunique()
-  last_user_timestamp = ratings_df[["userId", "timestamp"]].groupby(
-      "userId").max()
-
-  ratings_df["numberOfTimestamps"] = ratings_df["userId"].apply(
-      lambda x: num_timestamps["timestamp"][x])
-  ratings_df["lastTimestamp"] = ratings_df["userId"].apply(
-      lambda x: last_user_timestamp["timestamp"][x])
-
-  ratings_df = ratings_df[ratings_df["numberOfTimestamps"] > 2]
-
-  ratings_df = _create_row_col_indices(ratings_df)
-
-  train_ratings_df = ratings_df[
-      ratings_df["timestamp"] < ratings_df["lastTimestamp"]]
-  test_ratings_df = ratings_df[
-      ratings_df["timestamp"] == ratings_df["lastTimestamp"]]
-
-  return ratings_df, train_ratings_df, test_ratings_df
-
-
-def main(_):
-
-  # Fix seed for reproducibility
-  np.random.seed(FLAGS.random_seed)
-
-  logging.info("Loading MovieLens 20m from %s.", FLAGS.input_csv_file)
-  ratings_df = util.load_df_from_file(FLAGS.input_csv_file)
-  logging.info("Done loading MovieLens 20m from %s.", FLAGS.input_csv_file)
-
-  logging.info("Preprocessing MovieLens 20m.")
-  ratings_df, train_ratings_df, test_ratings_df = _preprocess_movie_lens(
-      ratings_df)
-  logging.info("Done preprocessing MovieLens 20m.")
-
-  num_users, num_items, _ = util.describe_rating_df(ratings_df, "original set")
-  _, _, num_train_ratings = util.describe_rating_df(
-      train_ratings_df, "train set")
-  _, _, num_test_ratings = util.describe_rating_df(test_ratings_df, "test set")
-
-  logging.info("Converting data frames to sparse matrices.")
-  train_ratings_matrix = util.convert_df_to_sparse_matrix(
-      train_ratings_df, shape=(num_users, num_items))
-  test_ratings_matrix = util.convert_df_to_sparse_matrix(
-      test_ratings_df, shape=(num_users, num_items))
-  logging.info("Done converting data frames to sparse matrices.")
-
-  reduced_num_rows = FLAGS.num_row_multiplier
-  reduced_num_cols = FLAGS.num_col_multiplier
-  k = min(reduced_num_rows, reduced_num_cols)
-  logging.info("Computing SVD of training matrix (top %d values).", k)
-  (u_train, s_train, v_train) = sparse_svd(
-      train_ratings_matrix, k, max_iter=None)
-  logging.info("Done computing SVD of training matrix.")
-
-  logging.info("Creating reduced rating matrix (size %d, %d)", reduced_num_rows,
-               reduced_num_cols)
-  reduced_train_matrix = resize_matrix(
-      (u_train, s_train, v_train), reduced_num_rows, reduced_num_cols)
-  reduced_train_matrix = normalize_matrix(reduced_train_matrix)
-  logging.info("Creating reduced rating matrix.")
-
-  average_sampling_rate = reduced_train_matrix.mean()
-  logging.info("Average sampling rate: %2f.", average_sampling_rate)
-  logging.info("Expected number of synthetic train samples: %s",
-               average_sampling_rate * num_train_ratings)
-  logging.info("Expected number of synthetic test samples: %s",
-               average_sampling_rate * num_test_ratings)
-
-  # Mark test data by a bit flip.
-  logging.info("Creating signed train/test matrix.")
-  train_test_ratings_matrix = train_ratings_matrix - test_ratings_matrix
-  train_test_ratings_matrix = train_test_ratings_matrix.tocoo()
-  logging.info("Done creating signed train/test matrix.")
-
-  output_train_file = (FLAGS.output_prefix + "trainx" + 
-      str(reduced_num_rows) + "x" + str(reduced_num_cols))
-  output_test_file = (FLAGS.output_prefix + "testx" +
-      str(reduced_num_rows) + "x" + str(reduced_num_cols))
-  output_train_file_metadata = None
-  output_test_file_metadata = None
-
-  logging.info("Creating synthetic train data set and dumping to %s.",
-               output_train_file)
-  logging.info("Creating synthetic train data set and dumping to %s.",
-               output_test_file)
-  output_randomized_kronecker_to_pickle(
-      left_matrix=reduced_train_matrix,
-      right_matrix=train_test_ratings_matrix,
-      train_indices_out_path=output_train_file,
-      test_indices_out_path=output_test_file,
-      train_metadata_out_path=output_train_file_metadata,
-      test_metadata_out_path=output_test_file_metadata)
-  logging.info("Done creating synthetic train data set and dumping to %s.",
-               output_train_file)
-  logging.info("Done creating synthetic test data set and dumping to %s.",
-               output_test_file)
-
-
-if __name__ == "__main__":
-  logging.set_verbosity(logging.INFO)
-  app.run(main)
diff --git a/data_generation/fractal_graph_expansions/util.py b/data_generation/fractal_graph_expansions/util.py
deleted file mode 100644
index 6b7ed901b..000000000
--- a/data_generation/fractal_graph_expansions/util.py
+++ /dev/null
@@ -1,103 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Miscellaneous utilities for fractal_graph_expansions.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from six.moves import cPickle as pickle
-
-
-
-from absl import logging
-
-import numpy as np
-import pandas as pd
-from scipy import sparse
-import tensorflow as tf
-
-
-def load_df_from_file(file_path, sep=",", header=0):
-  """Wrapper around pandas' read_csv."""
-  with tf.gfile.Open(file_path) as infile:
-    df = pd.read_csv(infile, sep=sep, header=header)
-  return df
-
-
-def convert_df_to_sparse_matrix(df, shape=None, row_name="row", col_name="col",
-                                data_name="data"):
-  row = df[row_name].values
-  col = df[col_name].values
-  data = df[data_name].values
-
-  return sparse.csr_matrix((data, (row, col)), shape=shape)
-
-
-def describe_rating_df(df, df_name=""):
-  num_ratings = len(df)
-  num_users = len(set(df["row"].values))
-  num_items = len(set(df["col"].values))
-
-  logging.info("%d users in ratings dataframe %s", num_users, df_name)
-  logging.info("%d items in ratings dataframe %s", num_items, df_name)
-  logging.info("%d ratings in ratings datagrame %s", num_ratings, df_name)
-
-  return num_users, num_items, num_ratings
-
-
-def serialize_to_file(obj, file_name, append=False):
-  """Pickle obj to file_name."""
-  logging.info("Serializing to file %s.", file_name)
-  with tf.gfile.Open(file_name, "a+" if append else "wb") as output_file:
-    pickle.dump(obj, output_file)
-  logging.info("Done serializing to file %s.", file_name)
-
-def savez_two_column(matrix, row_offset, file_name, append=False):
-  """Savez_compressed obj to file_name."""
-  logging.info("Saving obj to file in two column .npz format %s.", file_name)
-  tc = []
-  for u, items in enumerate(matrix):
-    user = row_offset + u
-    for item in items:
-      tc.append([user, item])
-  
-  np.savez_compressed(file_name, np.asarray(tc))
-  logging.info("Done saving to file %s.", file_name)
-
-def sorted_product_set(array_a, array_b):
-  """Compute the product set of array_a and array_b and sort it."""
-  return np.sort(
-      np.concatenate(
-          [array_a[i] * array_b for i in xrange(len(array_a))], axis=0)
-  )[::-1]
-
-
-def write_metadata_to_file(metadata_named_tuple, metadata_out_path, tag=""):
-  logging.info("Writing %s metadata file to %s", tag, metadata_out_path)
-  serialize_to_file(metadata_named_tuple, file_name=metadata_out_path)
-  logging.info("Done writing %s metadata file to %s", tag, metadata_out_path)
-
-
-def sparse_where_equal(coo_matrix, target_value):
-
-  cond_is_true = coo_matrix.data == target_value
-  data_where = coo_matrix.data[cond_is_true]
-  row_where = coo_matrix.row[cond_is_true]
-  col_where = coo_matrix.col[cond_is_true]
-
-  return sparse.csr_matrix(
-      (data_where, (row_where, col_where)), shape=coo_matrix.shape)
diff --git a/rnn_translator/.dockerignore b/retired_benchmarks/gnmt/.dockerignore
similarity index 100%
rename from rnn_translator/.dockerignore
rename to retired_benchmarks/gnmt/.dockerignore
diff --git a/rnn_translator/.gitignore b/retired_benchmarks/gnmt/.gitignore
similarity index 100%
rename from rnn_translator/.gitignore
rename to retired_benchmarks/gnmt/.gitignore
diff --git a/rnn_translator/README.md b/retired_benchmarks/gnmt/README.md
similarity index 100%
rename from rnn_translator/README.md
rename to retired_benchmarks/gnmt/README.md
diff --git a/rnn_translator/download_dataset.sh b/retired_benchmarks/gnmt/download_dataset.sh
similarity index 100%
rename from rnn_translator/download_dataset.sh
rename to retired_benchmarks/gnmt/download_dataset.sh
diff --git a/rnn_translator/pytorch/Dockerfile b/retired_benchmarks/gnmt/pytorch/Dockerfile
similarity index 100%
rename from rnn_translator/pytorch/Dockerfile
rename to retired_benchmarks/gnmt/pytorch/Dockerfile
diff --git a/rnn_translator/pytorch/LICENSE b/retired_benchmarks/gnmt/pytorch/LICENSE
similarity index 100%
rename from rnn_translator/pytorch/LICENSE
rename to retired_benchmarks/gnmt/pytorch/LICENSE
diff --git a/rnn_translator/pytorch/README.md b/retired_benchmarks/gnmt/pytorch/README.md
similarity index 100%
rename from rnn_translator/pytorch/README.md
rename to retired_benchmarks/gnmt/pytorch/README.md
diff --git a/rnn_translator/pytorch/requirements.txt b/retired_benchmarks/gnmt/pytorch/requirements.txt
similarity index 100%
rename from rnn_translator/pytorch/requirements.txt
rename to retired_benchmarks/gnmt/pytorch/requirements.txt
diff --git a/rnn_translator/pytorch/run.sh b/retired_benchmarks/gnmt/pytorch/run.sh
similarity index 100%
rename from rnn_translator/pytorch/run.sh
rename to retired_benchmarks/gnmt/pytorch/run.sh
diff --git a/rnn_translator/pytorch/run_and_time.sh b/retired_benchmarks/gnmt/pytorch/run_and_time.sh
similarity index 100%
rename from rnn_translator/pytorch/run_and_time.sh
rename to retired_benchmarks/gnmt/pytorch/run_and_time.sh
diff --git a/rnn_translator/pytorch/scripts/docker/build.sh b/retired_benchmarks/gnmt/pytorch/scripts/docker/build.sh
similarity index 100%
rename from rnn_translator/pytorch/scripts/docker/build.sh
rename to retired_benchmarks/gnmt/pytorch/scripts/docker/build.sh
diff --git a/rnn_translator/pytorch/scripts/docker/interactive.sh b/retired_benchmarks/gnmt/pytorch/scripts/docker/interactive.sh
similarity index 100%
rename from rnn_translator/pytorch/scripts/docker/interactive.sh
rename to retired_benchmarks/gnmt/pytorch/scripts/docker/interactive.sh
diff --git a/rnn_translator/pytorch/scripts/filter_dataset.py b/retired_benchmarks/gnmt/pytorch/scripts/filter_dataset.py
similarity index 100%
rename from rnn_translator/pytorch/scripts/filter_dataset.py
rename to retired_benchmarks/gnmt/pytorch/scripts/filter_dataset.py
diff --git a/rnn_translator/pytorch/seq2seq/data/config.py b/retired_benchmarks/gnmt/pytorch/seq2seq/data/config.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/data/config.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/data/config.py
diff --git a/rnn_translator/pytorch/seq2seq/data/dataset.py b/retired_benchmarks/gnmt/pytorch/seq2seq/data/dataset.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/data/dataset.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/data/dataset.py
diff --git a/rnn_translator/pytorch/seq2seq/data/sampler.py b/retired_benchmarks/gnmt/pytorch/seq2seq/data/sampler.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/data/sampler.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/data/sampler.py
diff --git a/rnn_translator/pytorch/seq2seq/data/tokenizer.py b/retired_benchmarks/gnmt/pytorch/seq2seq/data/tokenizer.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/data/tokenizer.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/data/tokenizer.py
diff --git a/rnn_translator/pytorch/seq2seq/inference/beam_search.py b/retired_benchmarks/gnmt/pytorch/seq2seq/inference/beam_search.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/inference/beam_search.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/inference/beam_search.py
diff --git a/rnn_translator/pytorch/seq2seq/inference/inference.py b/retired_benchmarks/gnmt/pytorch/seq2seq/inference/inference.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/inference/inference.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/inference/inference.py
diff --git a/rnn_translator/pytorch/seq2seq/models/attention.py b/retired_benchmarks/gnmt/pytorch/seq2seq/models/attention.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/models/attention.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/models/attention.py
diff --git a/rnn_translator/pytorch/seq2seq/models/decoder.py b/retired_benchmarks/gnmt/pytorch/seq2seq/models/decoder.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/models/decoder.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/models/decoder.py
diff --git a/rnn_translator/pytorch/seq2seq/models/encoder.py b/retired_benchmarks/gnmt/pytorch/seq2seq/models/encoder.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/models/encoder.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/models/encoder.py
diff --git a/rnn_translator/pytorch/seq2seq/models/gnmt.py b/retired_benchmarks/gnmt/pytorch/seq2seq/models/gnmt.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/models/gnmt.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/models/gnmt.py
diff --git a/rnn_translator/pytorch/seq2seq/models/seq2seq_base.py b/retired_benchmarks/gnmt/pytorch/seq2seq/models/seq2seq_base.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/models/seq2seq_base.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/models/seq2seq_base.py
diff --git a/rnn_translator/pytorch/seq2seq/train/fp_optimizers.py b/retired_benchmarks/gnmt/pytorch/seq2seq/train/fp_optimizers.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/train/fp_optimizers.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/train/fp_optimizers.py
diff --git a/rnn_translator/pytorch/seq2seq/train/lr_scheduler.py b/retired_benchmarks/gnmt/pytorch/seq2seq/train/lr_scheduler.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/train/lr_scheduler.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/train/lr_scheduler.py
diff --git a/rnn_translator/pytorch/seq2seq/train/smoothing.py b/retired_benchmarks/gnmt/pytorch/seq2seq/train/smoothing.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/train/smoothing.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/train/smoothing.py
diff --git a/rnn_translator/pytorch/seq2seq/train/trainer.py b/retired_benchmarks/gnmt/pytorch/seq2seq/train/trainer.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/train/trainer.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/train/trainer.py
diff --git a/rnn_translator/pytorch/seq2seq/utils.py b/retired_benchmarks/gnmt/pytorch/seq2seq/utils.py
similarity index 100%
rename from rnn_translator/pytorch/seq2seq/utils.py
rename to retired_benchmarks/gnmt/pytorch/seq2seq/utils.py
diff --git a/rnn_translator/pytorch/train.py b/retired_benchmarks/gnmt/pytorch/train.py
similarity index 100%
rename from rnn_translator/pytorch/train.py
rename to retired_benchmarks/gnmt/pytorch/train.py
diff --git a/rnn_translator/pytorch/translate.py b/retired_benchmarks/gnmt/pytorch/translate.py
similarity index 100%
rename from rnn_translator/pytorch/translate.py
rename to retired_benchmarks/gnmt/pytorch/translate.py
diff --git a/rnn_translator/verify_dataset.sh b/retired_benchmarks/gnmt/verify_dataset.sh
similarity index 100%
rename from rnn_translator/verify_dataset.sh
rename to retired_benchmarks/gnmt/verify_dataset.sh
diff --git a/recommendation/pytorch/.gitignore b/retired_benchmarks/ncf/.gitignore
similarity index 100%
rename from recommendation/pytorch/.gitignore
rename to retired_benchmarks/ncf/.gitignore
diff --git a/recommendation/pytorch/Dockerfile b/retired_benchmarks/ncf/Dockerfile
similarity index 100%
rename from recommendation/pytorch/Dockerfile
rename to retired_benchmarks/ncf/Dockerfile
diff --git a/recommendation/pytorch/README.md b/retired_benchmarks/ncf/README.md
similarity index 100%
rename from recommendation/pytorch/README.md
rename to retired_benchmarks/ncf/README.md
diff --git a/recommendation/pytorch/alias_generator.py b/retired_benchmarks/ncf/alias_generator.py
similarity index 100%
rename from recommendation/pytorch/alias_generator.py
rename to retired_benchmarks/ncf/alias_generator.py
diff --git a/recommendation/pytorch/convert.py b/retired_benchmarks/ncf/convert.py
similarity index 100%
rename from recommendation/pytorch/convert.py
rename to retired_benchmarks/ncf/convert.py
diff --git a/recommendation/pytorch/ncf.py b/retired_benchmarks/ncf/ncf.py
similarity index 100%
rename from recommendation/pytorch/ncf.py
rename to retired_benchmarks/ncf/ncf.py
diff --git a/recommendation/pytorch/negative_sampling_cpp/negative_sampling.cpp b/retired_benchmarks/ncf/negative_sampling_cpp/negative_sampling.cpp
similarity index 100%
rename from recommendation/pytorch/negative_sampling_cpp/negative_sampling.cpp
rename to retired_benchmarks/ncf/negative_sampling_cpp/negative_sampling.cpp
diff --git a/recommendation/pytorch/negative_sampling_cpp/setup.py b/retired_benchmarks/ncf/negative_sampling_cpp/setup.py
similarity index 100%
rename from recommendation/pytorch/negative_sampling_cpp/setup.py
rename to retired_benchmarks/ncf/negative_sampling_cpp/setup.py
diff --git a/recommendation/pytorch/neumf.py b/retired_benchmarks/ncf/neumf.py
similarity index 100%
rename from recommendation/pytorch/neumf.py
rename to retired_benchmarks/ncf/neumf.py
diff --git a/recommendation/pytorch/requirements.txt b/retired_benchmarks/ncf/requirements.txt
similarity index 100%
rename from recommendation/pytorch/requirements.txt
rename to retired_benchmarks/ncf/requirements.txt
diff --git a/recommendation/pytorch/run_and_time.sh b/retired_benchmarks/ncf/run_and_time.sh
similarity index 100%
rename from recommendation/pytorch/run_and_time.sh
rename to retired_benchmarks/ncf/run_and_time.sh
diff --git a/recommendation/pytorch/utils.py b/retired_benchmarks/ncf/utils.py
similarity index 100%
rename from recommendation/pytorch/utils.py
rename to retired_benchmarks/ncf/utils.py
diff --git a/sentiment_analysis/README.md b/retired_benchmarks/never-adopted/sentiment_analysis/README.md
similarity index 100%
rename from sentiment_analysis/README.md
rename to retired_benchmarks/never-adopted/sentiment_analysis/README.md
diff --git a/sentiment_analysis/download.py b/retired_benchmarks/never-adopted/sentiment_analysis/download.py
similarity index 100%
rename from sentiment_analysis/download.py
rename to retired_benchmarks/never-adopted/sentiment_analysis/download.py
diff --git a/sentiment_analysis/download_dataset.sh b/retired_benchmarks/never-adopted/sentiment_analysis/download_dataset.sh
similarity index 100%
rename from sentiment_analysis/download_dataset.sh
rename to retired_benchmarks/never-adopted/sentiment_analysis/download_dataset.sh
diff --git a/sentiment_analysis/paddle/run_and_time.sh b/retired_benchmarks/never-adopted/sentiment_analysis/paddle/run_and_time.sh
similarity index 100%
rename from sentiment_analysis/paddle/run_and_time.sh
rename to retired_benchmarks/never-adopted/sentiment_analysis/paddle/run_and_time.sh
diff --git a/sentiment_analysis/paddle/train.py b/retired_benchmarks/never-adopted/sentiment_analysis/paddle/train.py
similarity index 100%
rename from sentiment_analysis/paddle/train.py
rename to retired_benchmarks/never-adopted/sentiment_analysis/paddle/train.py
diff --git a/sentiment_analysis/verify.py b/retired_benchmarks/never-adopted/sentiment_analysis/verify.py
similarity index 100%
rename from sentiment_analysis/verify.py
rename to retired_benchmarks/never-adopted/sentiment_analysis/verify.py
diff --git a/sentiment_analysis/verify_dataset.sh b/retired_benchmarks/never-adopted/sentiment_analysis/verify_dataset.sh
similarity index 100%
rename from sentiment_analysis/verify_dataset.sh
rename to retired_benchmarks/never-adopted/sentiment_analysis/verify_dataset.sh
diff --git a/speech_recognition/README.md b/retired_benchmarks/never-adopted/speech_recognition/README.md
similarity index 100%
rename from speech_recognition/README.md
rename to retired_benchmarks/never-adopted/speech_recognition/README.md
diff --git a/speech_recognition/__init__.py b/retired_benchmarks/never-adopted/speech_recognition/__init__.py
similarity index 100%
rename from speech_recognition/__init__.py
rename to retired_benchmarks/never-adopted/speech_recognition/__init__.py
diff --git a/speech_recognition/data/.gitignore b/retired_benchmarks/never-adopted/speech_recognition/data/.gitignore
similarity index 100%
rename from speech_recognition/data/.gitignore
rename to retired_benchmarks/never-adopted/speech_recognition/data/.gitignore
diff --git a/speech_recognition/data/__init__.py b/retired_benchmarks/never-adopted/speech_recognition/data/__init__.py
similarity index 100%
rename from speech_recognition/data/__init__.py
rename to retired_benchmarks/never-adopted/speech_recognition/data/__init__.py
diff --git a/speech_recognition/data/bucketing_sampler.py b/retired_benchmarks/never-adopted/speech_recognition/data/bucketing_sampler.py
similarity index 100%
rename from speech_recognition/data/bucketing_sampler.py
rename to retired_benchmarks/never-adopted/speech_recognition/data/bucketing_sampler.py
diff --git a/speech_recognition/data/data-LibriSpeech-ref-cksum.out b/retired_benchmarks/never-adopted/speech_recognition/data/data-LibriSpeech-ref-cksum.out
similarity index 100%
rename from speech_recognition/data/data-LibriSpeech-ref-cksum.out
rename to retired_benchmarks/never-adopted/speech_recognition/data/data-LibriSpeech-ref-cksum.out
diff --git a/speech_recognition/data/data_loader.py b/retired_benchmarks/never-adopted/speech_recognition/data/data_loader.py
similarity index 100%
rename from speech_recognition/data/data_loader.py
rename to retired_benchmarks/never-adopted/speech_recognition/data/data_loader.py
diff --git a/speech_recognition/data/librispeech.py b/retired_benchmarks/never-adopted/speech_recognition/data/librispeech.py
similarity index 100%
rename from speech_recognition/data/librispeech.py
rename to retired_benchmarks/never-adopted/speech_recognition/data/librispeech.py
diff --git a/speech_recognition/data/merge_manifests.py b/retired_benchmarks/never-adopted/speech_recognition/data/merge_manifests.py
similarity index 100%
rename from speech_recognition/data/merge_manifests.py
rename to retired_benchmarks/never-adopted/speech_recognition/data/merge_manifests.py
diff --git a/speech_recognition/data/utils.py b/retired_benchmarks/never-adopted/speech_recognition/data/utils.py
similarity index 100%
rename from speech_recognition/data/utils.py
rename to retired_benchmarks/never-adopted/speech_recognition/data/utils.py
diff --git a/speech_recognition/download_dataset.sh b/retired_benchmarks/never-adopted/speech_recognition/download_dataset.sh
similarity index 100%
rename from speech_recognition/download_dataset.sh
rename to retired_benchmarks/never-adopted/speech_recognition/download_dataset.sh
diff --git a/speech_recognition/labels.json b/retired_benchmarks/never-adopted/speech_recognition/labels.json
similarity index 100%
rename from speech_recognition/labels.json
rename to retired_benchmarks/never-adopted/speech_recognition/labels.json
diff --git a/speech_recognition/pytorch/.gitignore b/retired_benchmarks/never-adopted/speech_recognition/pytorch/.gitignore
similarity index 100%
rename from speech_recognition/pytorch/.gitignore
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/.gitignore
diff --git a/speech_recognition/pytorch/decoder.py b/retired_benchmarks/never-adopted/speech_recognition/pytorch/decoder.py
similarity index 100%
rename from speech_recognition/pytorch/decoder.py
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/decoder.py
diff --git a/speech_recognition/pytorch/docker/Dockerfile.gpu b/retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/Dockerfile.gpu
similarity index 100%
rename from speech_recognition/pytorch/docker/Dockerfile.gpu
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/Dockerfile.gpu
diff --git a/speech_recognition/pytorch/docker/base.gpu b/retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/base.gpu
similarity index 100%
rename from speech_recognition/pytorch/docker/base.gpu
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/base.gpu
diff --git a/speech_recognition/pytorch/docker/build-docker.sh b/retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/build-docker.sh
similarity index 100%
rename from speech_recognition/pytorch/docker/build-docker.sh
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/build-docker.sh
diff --git a/speech_recognition/pytorch/docker/run-dev.sh b/retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/run-dev.sh
similarity index 100%
rename from speech_recognition/pytorch/docker/run-dev.sh
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/docker/run-dev.sh
diff --git a/speech_recognition/pytorch/eval_model.py b/retired_benchmarks/never-adopted/speech_recognition/pytorch/eval_model.py
similarity index 100%
rename from speech_recognition/pytorch/eval_model.py
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/eval_model.py
diff --git a/speech_recognition/pytorch/model.py b/retired_benchmarks/never-adopted/speech_recognition/pytorch/model.py
similarity index 100%
rename from speech_recognition/pytorch/model.py
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/model.py
diff --git a/speech_recognition/pytorch/params.py b/retired_benchmarks/never-adopted/speech_recognition/pytorch/params.py
similarity index 100%
rename from speech_recognition/pytorch/params.py
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/params.py
diff --git a/speech_recognition/pytorch/run_and_time.sh b/retired_benchmarks/never-adopted/speech_recognition/pytorch/run_and_time.sh
similarity index 100%
rename from speech_recognition/pytorch/run_and_time.sh
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/run_and_time.sh
diff --git a/speech_recognition/pytorch/train.py b/retired_benchmarks/never-adopted/speech_recognition/pytorch/train.py
similarity index 100%
rename from speech_recognition/pytorch/train.py
rename to retired_benchmarks/never-adopted/speech_recognition/pytorch/train.py
diff --git a/speech_recognition/verify_dataset.sh b/retired_benchmarks/never-adopted/speech_recognition/verify_dataset.sh
similarity index 100%
rename from speech_recognition/verify_dataset.sh
rename to retired_benchmarks/never-adopted/speech_recognition/verify_dataset.sh
diff --git a/image_classification/tensorflow/Dockerfile b/retired_benchmarks/resnet-tf1/Dockerfile
similarity index 100%
rename from image_classification/tensorflow/Dockerfile
rename to retired_benchmarks/resnet-tf1/Dockerfile
diff --git a/image_classification/tensorflow/README.md b/retired_benchmarks/resnet-tf1/README.md
similarity index 100%
rename from image_classification/tensorflow/README.md
rename to retired_benchmarks/resnet-tf1/README.md
diff --git a/image_classification/tensorflow/log_stitch.py b/retired_benchmarks/resnet-tf1/log_stitch.py
similarity index 100%
rename from image_classification/tensorflow/log_stitch.py
rename to retired_benchmarks/resnet-tf1/log_stitch.py
diff --git a/image_classification/tensorflow/official/.gitignore b/retired_benchmarks/resnet-tf1/official/.gitignore
similarity index 100%
rename from image_classification/tensorflow/official/.gitignore
rename to retired_benchmarks/resnet-tf1/official/.gitignore
diff --git a/image_classification/tensorflow/official/Dockerfile.cpu b/retired_benchmarks/resnet-tf1/official/Dockerfile.cpu
similarity index 100%
rename from image_classification/tensorflow/official/Dockerfile.cpu
rename to retired_benchmarks/resnet-tf1/official/Dockerfile.cpu
diff --git a/image_classification/tensorflow/official/Dockerfile.gpu b/retired_benchmarks/resnet-tf1/official/Dockerfile.gpu
similarity index 100%
rename from image_classification/tensorflow/official/Dockerfile.gpu
rename to retired_benchmarks/resnet-tf1/official/Dockerfile.gpu
diff --git a/image_classification/tensorflow/official/README.md b/retired_benchmarks/resnet-tf1/official/README.md
similarity index 100%
rename from image_classification/tensorflow/official/README.md
rename to retired_benchmarks/resnet-tf1/official/README.md
diff --git a/compliance/verify_submission/mlperf_submission_helper/__init__.py b/retired_benchmarks/resnet-tf1/official/__init__.py
similarity index 100%
rename from compliance/verify_submission/mlperf_submission_helper/__init__.py
rename to retired_benchmarks/resnet-tf1/official/__init__.py
diff --git a/image_classification/tensorflow/official/benchmark/datastore/schema/benchmark_metric.json b/retired_benchmarks/resnet-tf1/official/benchmark/datastore/schema/benchmark_metric.json
similarity index 100%
rename from image_classification/tensorflow/official/benchmark/datastore/schema/benchmark_metric.json
rename to retired_benchmarks/resnet-tf1/official/benchmark/datastore/schema/benchmark_metric.json
diff --git a/image_classification/tensorflow/official/benchmark/datastore/schema/benchmark_run.json b/retired_benchmarks/resnet-tf1/official/benchmark/datastore/schema/benchmark_run.json
similarity index 100%
rename from image_classification/tensorflow/official/benchmark/datastore/schema/benchmark_run.json
rename to retired_benchmarks/resnet-tf1/official/benchmark/datastore/schema/benchmark_run.json
diff --git a/image_classification/tensorflow/official/requirements.txt b/retired_benchmarks/resnet-tf1/official/requirements.txt
similarity index 100%
rename from image_classification/tensorflow/official/requirements.txt
rename to retired_benchmarks/resnet-tf1/official/requirements.txt
diff --git a/image_classification/tensorflow/official/resnet/README.md b/retired_benchmarks/resnet-tf1/official/resnet/README.md
similarity index 100%
rename from image_classification/tensorflow/official/resnet/README.md
rename to retired_benchmarks/resnet-tf1/official/resnet/README.md
diff --git a/data_generation/__init__.py b/retired_benchmarks/resnet-tf1/official/resnet/__init__.py
similarity index 100%
rename from data_generation/__init__.py
rename to retired_benchmarks/resnet-tf1/official/resnet/__init__.py
diff --git a/image_classification/tensorflow/official/resnet/imagenet_main.py b/retired_benchmarks/resnet-tf1/official/resnet/imagenet_main.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/imagenet_main.py
rename to retired_benchmarks/resnet-tf1/official/resnet/imagenet_main.py
diff --git a/image_classification/tensorflow/official/resnet/imagenet_preprocessing.py b/retired_benchmarks/resnet-tf1/official/resnet/imagenet_preprocessing.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/imagenet_preprocessing.py
rename to retired_benchmarks/resnet-tf1/official/resnet/imagenet_preprocessing.py
diff --git a/image_classification/tensorflow/official/resnet/resnet_model.py b/retired_benchmarks/resnet-tf1/official/resnet/resnet_model.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/resnet_model.py
rename to retired_benchmarks/resnet-tf1/official/resnet/resnet_model.py
diff --git a/image_classification/tensorflow/official/resnet/resnet_run_loop.py b/retired_benchmarks/resnet-tf1/official/resnet/resnet_run_loop.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/resnet_run_loop.py
rename to retired_benchmarks/resnet-tf1/official/resnet/resnet_run_loop.py
diff --git a/data_generation/fractal_graph_expansions/__init__.py b/retired_benchmarks/resnet-tf1/official/utils/__init__.py
similarity index 100%
rename from data_generation/fractal_graph_expansions/__init__.py
rename to retired_benchmarks/resnet-tf1/official/utils/__init__.py
diff --git a/image_classification/tensorflow/official/__init__.py b/retired_benchmarks/resnet-tf1/official/utils/arg_parsers/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/__init__.py
rename to retired_benchmarks/resnet-tf1/official/utils/arg_parsers/__init__.py
diff --git a/image_classification/tensorflow/official/utils/arg_parsers/parsers.py b/retired_benchmarks/resnet-tf1/official/utils/arg_parsers/parsers.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/arg_parsers/parsers.py
rename to retired_benchmarks/resnet-tf1/official/utils/arg_parsers/parsers.py
diff --git a/image_classification/tensorflow/official/resnet/__init__.py b/retired_benchmarks/resnet-tf1/official/utils/export/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/__init__.py
rename to retired_benchmarks/resnet-tf1/official/utils/export/__init__.py
diff --git a/image_classification/tensorflow/official/utils/export/export.py b/retired_benchmarks/resnet-tf1/official/utils/export/export.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/export/export.py
rename to retired_benchmarks/resnet-tf1/official/utils/export/export.py
diff --git a/image_classification/tensorflow/official/utils/__init__.py b/retired_benchmarks/resnet-tf1/official/utils/logs/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/__init__.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/__init__.py
diff --git a/image_classification/tensorflow/official/utils/logs/benchmark_uploader.py b/retired_benchmarks/resnet-tf1/official/utils/logs/benchmark_uploader.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/benchmark_uploader.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/benchmark_uploader.py
diff --git a/image_classification/tensorflow/official/utils/logs/hooks.py b/retired_benchmarks/resnet-tf1/official/utils/logs/hooks.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/hooks.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/hooks.py
diff --git a/image_classification/tensorflow/official/utils/logs/hooks_helper.py b/retired_benchmarks/resnet-tf1/official/utils/logs/hooks_helper.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/hooks_helper.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/hooks_helper.py
diff --git a/image_classification/tensorflow/official/utils/logs/logger.py b/retired_benchmarks/resnet-tf1/official/utils/logs/logger.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/logger.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/logger.py
diff --git a/image_classification/tensorflow/official/utils/logs/metric_hook.py b/retired_benchmarks/resnet-tf1/official/utils/logs/metric_hook.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/metric_hook.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/metric_hook.py
diff --git a/image_classification/tensorflow/official/utils/arg_parsers/__init__.py b/retired_benchmarks/resnet-tf1/official/utils/misc/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/arg_parsers/__init__.py
rename to retired_benchmarks/resnet-tf1/official/utils/misc/__init__.py
diff --git a/image_classification/tensorflow/official/utils/misc/model_helpers.py b/retired_benchmarks/resnet-tf1/official/utils/misc/model_helpers.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/misc/model_helpers.py
rename to retired_benchmarks/resnet-tf1/official/utils/misc/model_helpers.py
diff --git a/image_classification/tensorflow/official_diff.txt b/retired_benchmarks/resnet-tf1/official_diff.txt
similarity index 100%
rename from image_classification/tensorflow/official_diff.txt
rename to retired_benchmarks/resnet-tf1/official_diff.txt
diff --git a/image_classification/tensorflow/preprocess.sh b/retired_benchmarks/resnet-tf1/preprocess.sh
similarity index 100%
rename from image_classification/tensorflow/preprocess.sh
rename to retired_benchmarks/resnet-tf1/preprocess.sh
diff --git a/image_classification/tensorflow/requirements.txt b/retired_benchmarks/resnet-tf1/requirements.txt
similarity index 100%
rename from image_classification/tensorflow/requirements.txt
rename to retired_benchmarks/resnet-tf1/requirements.txt
diff --git a/image_classification/tensorflow/run.sh b/retired_benchmarks/resnet-tf1/run.sh
similarity index 100%
rename from image_classification/tensorflow/run.sh
rename to retired_benchmarks/resnet-tf1/run.sh
diff --git a/image_classification/tensorflow/run_and_time.sh b/retired_benchmarks/resnet-tf1/run_and_time.sh
similarity index 100%
rename from image_classification/tensorflow/run_and_time.sh
rename to retired_benchmarks/resnet-tf1/run_and_time.sh
diff --git a/retired_benchmarks/single_stage_detector/Dockerfile b/retired_benchmarks/ssd-v1/Dockerfile
similarity index 100%
rename from retired_benchmarks/single_stage_detector/Dockerfile
rename to retired_benchmarks/ssd-v1/Dockerfile
diff --git a/retired_benchmarks/single_stage_detector/download_dataset.sh b/retired_benchmarks/ssd-v1/download_dataset.sh
similarity index 100%
rename from retired_benchmarks/single_stage_detector/download_dataset.sh
rename to retired_benchmarks/ssd-v1/download_dataset.sh
diff --git a/retired_benchmarks/single_stage_detector/download_resnet34_backbone.sh b/retired_benchmarks/ssd-v1/download_resnet34_backbone.sh
similarity index 100%
rename from retired_benchmarks/single_stage_detector/download_resnet34_backbone.sh
rename to retired_benchmarks/ssd-v1/download_resnet34_backbone.sh
diff --git a/retired_benchmarks/single_stage_detector/pth_to_pickle.py b/retired_benchmarks/ssd-v1/pth_to_pickle.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/pth_to_pickle.py
rename to retired_benchmarks/ssd-v1/pth_to_pickle.py
diff --git a/retired_benchmarks/single_stage_detector/requirements.txt b/retired_benchmarks/ssd-v1/requirements.txt
similarity index 100%
rename from retired_benchmarks/single_stage_detector/requirements.txt
rename to retired_benchmarks/ssd-v1/requirements.txt
diff --git a/retired_benchmarks/single_stage_detector/ssd/README.md b/retired_benchmarks/ssd-v1/ssd/README.md
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/README.md
rename to retired_benchmarks/ssd-v1/ssd/README.md
diff --git a/retired_benchmarks/single_stage_detector/ssd/base_model.py b/retired_benchmarks/ssd-v1/ssd/base_model.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/base_model.py
rename to retired_benchmarks/ssd-v1/ssd/base_model.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/bind_launch.py b/retired_benchmarks/ssd-v1/ssd/bind_launch.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/bind_launch.py
rename to retired_benchmarks/ssd-v1/ssd/bind_launch.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/coco.py b/retired_benchmarks/ssd-v1/ssd/coco.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/coco.py
rename to retired_benchmarks/ssd-v1/ssd/coco.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/config_DGX1_32.sh b/retired_benchmarks/ssd-v1/ssd/config_DGX1_32.sh
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/config_DGX1_32.sh
rename to retired_benchmarks/ssd-v1/ssd/config_DGX1_32.sh
diff --git a/retired_benchmarks/single_stage_detector/ssd/config_DGX1_multinode.sh b/retired_benchmarks/ssd-v1/ssd/config_DGX1_multinode.sh
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/config_DGX1_multinode.sh
rename to retired_benchmarks/ssd-v1/ssd/config_DGX1_multinode.sh
diff --git a/retired_benchmarks/single_stage_detector/ssd/config_DGX1_singlenode.sh b/retired_benchmarks/ssd-v1/ssd/config_DGX1_singlenode.sh
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/config_DGX1_singlenode.sh
rename to retired_benchmarks/ssd-v1/ssd/config_DGX1_singlenode.sh
diff --git a/retired_benchmarks/single_stage_detector/ssd/distributed.py b/retired_benchmarks/ssd-v1/ssd/distributed.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/distributed.py
rename to retired_benchmarks/ssd-v1/ssd/distributed.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/eval.py b/retired_benchmarks/ssd-v1/ssd/eval.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/eval.py
rename to retired_benchmarks/ssd-v1/ssd/eval.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/mlperf_logger.py b/retired_benchmarks/ssd-v1/ssd/mlperf_logger.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/mlperf_logger.py
rename to retired_benchmarks/ssd-v1/ssd/mlperf_logger.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/run.sub b/retired_benchmarks/ssd-v1/ssd/run.sub
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/run.sub
rename to retired_benchmarks/ssd-v1/ssd/run.sub
diff --git a/retired_benchmarks/single_stage_detector/ssd/run_and_time.sh b/retired_benchmarks/ssd-v1/ssd/run_and_time.sh
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/run_and_time.sh
rename to retired_benchmarks/ssd-v1/ssd/run_and_time.sh
diff --git a/retired_benchmarks/single_stage_detector/ssd/ssd300.py b/retired_benchmarks/ssd-v1/ssd/ssd300.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/ssd300.py
rename to retired_benchmarks/ssd-v1/ssd/ssd300.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/train.py b/retired_benchmarks/ssd-v1/ssd/train.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/train.py
rename to retired_benchmarks/ssd-v1/ssd/train.py
diff --git a/retired_benchmarks/single_stage_detector/ssd/utils.py b/retired_benchmarks/ssd-v1/ssd/utils.py
similarity index 100%
rename from retired_benchmarks/single_stage_detector/ssd/utils.py
rename to retired_benchmarks/ssd-v1/ssd/utils.py
diff --git a/translation/README.md b/retired_benchmarks/transformer/README.md
similarity index 100%
rename from translation/README.md
rename to retired_benchmarks/transformer/README.md
diff --git a/translation/data_download.py b/retired_benchmarks/transformer/data_download.py
similarity index 100%
rename from translation/data_download.py
rename to retired_benchmarks/transformer/data_download.py
diff --git a/translation/download_data.sh b/retired_benchmarks/transformer/download_data.sh
similarity index 100%
rename from translation/download_data.sh
rename to retired_benchmarks/transformer/download_data.sh
diff --git a/translation/tensorflow/Dockerfile b/retired_benchmarks/transformer/tensorflow/Dockerfile
similarity index 100%
rename from translation/tensorflow/Dockerfile
rename to retired_benchmarks/transformer/tensorflow/Dockerfile
diff --git a/translation/tensorflow/bert/README.md b/retired_benchmarks/transformer/tensorflow/bert/README.md
similarity index 100%
rename from translation/tensorflow/bert/README.md
rename to retired_benchmarks/transformer/tensorflow/bert/README.md
diff --git a/translation/tensorflow/bert/__init__.py b/retired_benchmarks/transformer/tensorflow/bert/__init__.py
similarity index 100%
rename from translation/tensorflow/bert/__init__.py
rename to retired_benchmarks/transformer/tensorflow/bert/__init__.py
diff --git a/translation/tensorflow/bert/create_pretraining_data.py b/retired_benchmarks/transformer/tensorflow/bert/create_pretraining_data.py
similarity index 100%
rename from translation/tensorflow/bert/create_pretraining_data.py
rename to retired_benchmarks/transformer/tensorflow/bert/create_pretraining_data.py
diff --git a/translation/tensorflow/bert/extract_features.py b/retired_benchmarks/transformer/tensorflow/bert/extract_features.py
similarity index 100%
rename from translation/tensorflow/bert/extract_features.py
rename to retired_benchmarks/transformer/tensorflow/bert/extract_features.py
diff --git a/translation/tensorflow/bert/lamb_optimizer_v1.py b/retired_benchmarks/transformer/tensorflow/bert/lamb_optimizer_v1.py
similarity index 100%
rename from translation/tensorflow/bert/lamb_optimizer_v1.py
rename to retired_benchmarks/transformer/tensorflow/bert/lamb_optimizer_v1.py
diff --git a/translation/tensorflow/bert/modeling.py b/retired_benchmarks/transformer/tensorflow/bert/modeling.py
similarity index 100%
rename from translation/tensorflow/bert/modeling.py
rename to retired_benchmarks/transformer/tensorflow/bert/modeling.py
diff --git a/translation/tensorflow/bert/optimization.py b/retired_benchmarks/transformer/tensorflow/bert/optimization.py
similarity index 100%
rename from translation/tensorflow/bert/optimization.py
rename to retired_benchmarks/transformer/tensorflow/bert/optimization.py
diff --git a/translation/tensorflow/bert/run_classifier.py b/retired_benchmarks/transformer/tensorflow/bert/run_classifier.py
similarity index 100%
rename from translation/tensorflow/bert/run_classifier.py
rename to retired_benchmarks/transformer/tensorflow/bert/run_classifier.py
diff --git a/translation/tensorflow/bert/run_pretraining.py b/retired_benchmarks/transformer/tensorflow/bert/run_pretraining.py
similarity index 100%
rename from translation/tensorflow/bert/run_pretraining.py
rename to retired_benchmarks/transformer/tensorflow/bert/run_pretraining.py
diff --git a/translation/tensorflow/bert/run_squad.py b/retired_benchmarks/transformer/tensorflow/bert/run_squad.py
similarity index 100%
rename from translation/tensorflow/bert/run_squad.py
rename to retired_benchmarks/transformer/tensorflow/bert/run_squad.py
diff --git a/translation/tensorflow/bert/tokenization.py b/retired_benchmarks/transformer/tensorflow/bert/tokenization.py
similarity index 100%
rename from translation/tensorflow/bert/tokenization.py
rename to retired_benchmarks/transformer/tensorflow/bert/tokenization.py
diff --git a/translation/tensorflow/process_data.py b/retired_benchmarks/transformer/tensorflow/process_data.py
similarity index 100%
rename from translation/tensorflow/process_data.py
rename to retired_benchmarks/transformer/tensorflow/process_data.py
diff --git a/translation/tensorflow/requirements.txt b/retired_benchmarks/transformer/tensorflow/requirements.txt
similarity index 100%
rename from translation/tensorflow/requirements.txt
rename to retired_benchmarks/transformer/tensorflow/requirements.txt
diff --git a/translation/tensorflow/run_and_time.sh b/retired_benchmarks/transformer/tensorflow/run_and_time.sh
similarity index 100%
rename from translation/tensorflow/run_and_time.sh
rename to retired_benchmarks/transformer/tensorflow/run_and_time.sh
diff --git a/translation/tensorflow/run_preprocessing.sh b/retired_benchmarks/transformer/tensorflow/run_preprocessing.sh
similarity index 100%
rename from translation/tensorflow/run_preprocessing.sh
rename to retired_benchmarks/transformer/tensorflow/run_preprocessing.sh
diff --git a/translation/tensorflow/run_training.sh b/retired_benchmarks/transformer/tensorflow/run_training.sh
similarity index 100%
rename from translation/tensorflow/run_training.sh
rename to retired_benchmarks/transformer/tensorflow/run_training.sh
diff --git a/translation/tensorflow/transformer/README.md b/retired_benchmarks/transformer/tensorflow/transformer/README.md
similarity index 100%
rename from translation/tensorflow/transformer/README.md
rename to retired_benchmarks/transformer/tensorflow/transformer/README.md
diff --git a/image_classification/tensorflow/official/utils/logs/__init__.py b/retired_benchmarks/transformer/tensorflow/transformer/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/__init__.py
rename to retired_benchmarks/transformer/tensorflow/transformer/__init__.py
diff --git a/translation/tensorflow/transformer/compute_bleu.py b/retired_benchmarks/transformer/tensorflow/transformer/compute_bleu.py
similarity index 100%
rename from translation/tensorflow/transformer/compute_bleu.py
rename to retired_benchmarks/transformer/tensorflow/transformer/compute_bleu.py
diff --git a/translation/tensorflow/transformer/data_download.py b/retired_benchmarks/transformer/tensorflow/transformer/data_download.py
similarity index 100%
rename from translation/tensorflow/transformer/data_download.py
rename to retired_benchmarks/transformer/tensorflow/transformer/data_download.py
diff --git a/image_classification/tensorflow/official/utils/misc/__init__.py b/retired_benchmarks/transformer/tensorflow/transformer/model/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/misc/__init__.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/__init__.py
diff --git a/translation/tensorflow/transformer/model/attention_layer.py b/retired_benchmarks/transformer/tensorflow/transformer/model/attention_layer.py
similarity index 100%
rename from translation/tensorflow/transformer/model/attention_layer.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/attention_layer.py
diff --git a/translation/tensorflow/transformer/model/beam_search.py b/retired_benchmarks/transformer/tensorflow/transformer/model/beam_search.py
similarity index 100%
rename from translation/tensorflow/transformer/model/beam_search.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/beam_search.py
diff --git a/translation/tensorflow/transformer/model/embedding_layer.py b/retired_benchmarks/transformer/tensorflow/transformer/model/embedding_layer.py
similarity index 100%
rename from translation/tensorflow/transformer/model/embedding_layer.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/embedding_layer.py
diff --git a/translation/tensorflow/transformer/model/ffn_layer.py b/retired_benchmarks/transformer/tensorflow/transformer/model/ffn_layer.py
similarity index 100%
rename from translation/tensorflow/transformer/model/ffn_layer.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/ffn_layer.py
diff --git a/translation/tensorflow/transformer/model/model_params.py b/retired_benchmarks/transformer/tensorflow/transformer/model/model_params.py
similarity index 100%
rename from translation/tensorflow/transformer/model/model_params.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/model_params.py
diff --git a/translation/tensorflow/transformer/model/model_utils.py b/retired_benchmarks/transformer/tensorflow/transformer/model/model_utils.py
similarity index 100%
rename from translation/tensorflow/transformer/model/model_utils.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/model_utils.py
diff --git a/translation/tensorflow/transformer/model/transformer.py b/retired_benchmarks/transformer/tensorflow/transformer/model/transformer.py
similarity index 100%
rename from translation/tensorflow/transformer/model/transformer.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/transformer.py
diff --git a/translation/tensorflow/transformer/transformer_main.py b/retired_benchmarks/transformer/tensorflow/transformer/transformer_main.py
similarity index 100%
rename from translation/tensorflow/transformer/transformer_main.py
rename to retired_benchmarks/transformer/tensorflow/transformer/transformer_main.py
diff --git a/translation/tensorflow/transformer/translate.py b/retired_benchmarks/transformer/tensorflow/transformer/translate.py
similarity index 100%
rename from translation/tensorflow/transformer/translate.py
rename to retired_benchmarks/transformer/tensorflow/transformer/translate.py
diff --git a/image_classification/tensorflow/official/utils/testing/__init__.py b/retired_benchmarks/transformer/tensorflow/transformer/utils/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/__init__.py
rename to retired_benchmarks/transformer/tensorflow/transformer/utils/__init__.py
diff --git a/translation/tensorflow/transformer/utils/dataset.py b/retired_benchmarks/transformer/tensorflow/transformer/utils/dataset.py
similarity index 100%
rename from translation/tensorflow/transformer/utils/dataset.py
rename to retired_benchmarks/transformer/tensorflow/transformer/utils/dataset.py
diff --git a/translation/tensorflow/transformer/utils/metrics.py b/retired_benchmarks/transformer/tensorflow/transformer/utils/metrics.py
similarity index 100%
rename from translation/tensorflow/transformer/utils/metrics.py
rename to retired_benchmarks/transformer/tensorflow/transformer/utils/metrics.py
diff --git a/translation/tensorflow/transformer/utils/tokenizer.py b/retired_benchmarks/transformer/tensorflow/transformer/utils/tokenizer.py
similarity index 100%
rename from translation/tensorflow/transformer/utils/tokenizer.py
rename to retired_benchmarks/transformer/tensorflow/transformer/utils/tokenizer.py
diff --git a/translation/tensorflow/transformer/vocab/vocab.translate_ende_wmt32k.32768.subwords b/retired_benchmarks/transformer/tensorflow/transformer/vocab/vocab.translate_ende_wmt32k.32768.subwords
similarity index 100%
rename from translation/tensorflow/transformer/vocab/vocab.translate_ende_wmt32k.32768.subwords
rename to retired_benchmarks/transformer/tensorflow/transformer/vocab/vocab.translate_ende_wmt32k.32768.subwords
diff --git a/translation/tensorflow/transformer_diff.txt b/retired_benchmarks/transformer/tensorflow/transformer_diff.txt
similarity index 100%
rename from translation/tensorflow/transformer_diff.txt
rename to retired_benchmarks/transformer/tensorflow/transformer_diff.txt
diff --git a/translation/verify_dataset.sh b/retired_benchmarks/transformer/verify_dataset.sh
similarity index 100%
rename from translation/verify_dataset.sh
rename to retired_benchmarks/transformer/verify_dataset.sh
diff --git a/translation/tensorflow/transformer/__init__.py b/translation/tensorflow/transformer/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/translation/tensorflow/transformer/model/__init__.py b/translation/tensorflow/transformer/model/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/translation/tensorflow/transformer/utils/__init__.py b/translation/tensorflow/transformer/utils/__init__.py
deleted file mode 100644
index e69de29bb..000000000


Test patch:
diff --git a/compliance/mllog/test_mllog.py b/compliance/mllog/test_mllog.py
deleted file mode 100644
index 41d771035..000000000
--- a/compliance/mllog/test_mllog.py
+++ /dev/null
@@ -1,134 +0,0 @@
-# Copyright 2019 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-import collections
-from contextlib import contextmanager
-import io
-import json
-import os
-import sys
-import time
-import unittest
-from unittest import mock
-
-import mllog
-from mllog import constants
-
-
-@contextmanager
-def _captured_stdout():
-  cap_out = io.StringIO()
-  old_out = sys.stdout
-  try:
-    sys.stdout = cap_out
-    yield sys.stdout
-  finally:
-    sys.stdout = old_out
-
-class TestMlperfLog(unittest.TestCase):
-
-  def setUp(self):
-    self.origin_get_caller = mllog.mllog.get_caller
-    mllog.mllog.get_caller = mock.MagicMock(
-        return_value={"file": "mybenchmark/file.py", "lineno": 42})
-
-    self.origin_do_log = mllog.mllog.MLLogger._do_log
-    mllog.mllog.MLLogger._do_log = mock.Mock(side_effect=self._fake_do_log)
-
-    self.origin_time = time.time
-    time.time = mock.MagicMock(return_value=1234567890.123)
-
-  def tearDown(self):
-    mllog.mllog.get_caller = self.origin_get_caller
-    time.time = self.origin_time
-
-  def _fake_do_log(self, level, message, clear_line=False):
-    if clear_line:
-      print("\n" + message)
-    else:
-      print(message)
-
-  def test_mllog_start_simple(self):
-    prefix = ":::MLLOG"
-    expected_log_json = json.dumps(json.loads(r'''
-        {
-          "namespace": "",
-          "time_ms": 1234567890123,
-          "event_type": "INTERVAL_START",
-          "key": "run_start",
-          "value": null,
-          "metadata": {"file": "mybenchmark/file.py", "lineno": 42}
-        }''', object_pairs_hook=collections.OrderedDict))
-    expected_output = " ".join([prefix, expected_log_json])
-    with _captured_stdout() as out:
-      mllogger = mllog.get_mllogger()
-      mllogger.start(constants.RUN_START, None)
-      self.assertEqual(out.getvalue().splitlines()[0], expected_output)
-
-  def test_mllog_end_simple(self):
-    prefix = ":::MLLOG"
-    expected_log_json = json.dumps(json.loads(r'''
-        {
-          "namespace": "",
-          "time_ms": 1234567890123,
-          "event_type": "INTERVAL_END",
-          "key": "run_stop",
-          "value": null,
-          "metadata": {"file": "mybenchmark/file.py", "lineno": 42}
-        }''', object_pairs_hook=collections.OrderedDict))
-    expected_output = " ".join([prefix, expected_log_json])
-    with _captured_stdout() as out:
-      mllogger = mllog.get_mllogger()
-      mllogger.end(constants.RUN_STOP, None)
-      self.assertEqual(out.getvalue().splitlines()[0], expected_output)
-
-  def test_mllog_event_simple(self):
-    prefix = ":::MLLOG"
-    expected_log_json = json.dumps(json.loads(r'''
-        {
-          "namespace": "",
-          "time_ms": 1234567890123,
-          "event_type": "POINT_IN_TIME",
-          "key": "eval_accuracy",
-          "value": 0.99,
-          "metadata": {"file": "mybenchmark/file.py", "lineno": 42}
-        }''', object_pairs_hook=collections.OrderedDict))
-    expected_output = " ".join([prefix, expected_log_json])
-    with _captured_stdout() as out:
-      mllogger = mllog.get_mllogger()
-      mllogger.event(constants.EVAL_ACCURACY, 0.99)
-      self.assertEqual(out.getvalue().splitlines()[0], expected_output)
-
-  def test_mllog_event_override_param(self):
-    prefix = ":::MLLOG"
-    expected_log_json = json.dumps(json.loads(r'''
-        {
-          "namespace": "worker1",
-          "time_ms": 1231231230123,
-          "event_type": "POINT_IN_TIME",
-          "key": "eval_accuracy",
-          "value": 0.99,
-          "metadata": {"file": "mybenchmark/file.py", "lineno": 42}
-        }''', object_pairs_hook=collections.OrderedDict))
-    expected_output = "\n" + " ".join([prefix, expected_log_json]) + "\n"
-    with _captured_stdout() as out:
-      mllogger = mllog.get_mllogger()
-      mllogger.event(constants.EVAL_ACCURACY, 0.99, namespace="worker1",
-                     time_ms=1231231230123, clear_line=True)
-      self.assertEqual(out.getvalue(), expected_output)
-
-
-if __name__ == "__main__":
-  unittest.main()
diff --git a/compliance/mlperf_compliance/test_tag_set.py b/compliance/mlperf_compliance/test_tag_set.py
deleted file mode 100644
index 7c6bcb4af..000000000
--- a/compliance/mlperf_compliance/test_tag_set.py
+++ /dev/null
@@ -1,69 +0,0 @@
-# Copyright 2018 MLBenchmark Group. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Verification script to check model tag sets.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from collections import defaultdict
-import re
-
-from mlperf_compliance import _gnmt_tags
-from mlperf_compliance import _ncf_tags
-from mlperf_compliance import _resnet_tags
-from mlperf_compliance import _ssd_tags
-from mlperf_compliance import _transformer_tags
-from mlperf_compliance  import tags
-
-
-_MODEL_TAG_MODULES = [_gnmt_tags, _ncf_tags, _resnet_tags, _ssd_tags,
-                      _transformer_tags]
-
-TAG_PATTERN = re.compile("^[A-Za-z0-9_]+$")
-
-
-def extract_tags(module):
-  output = []
-  for i in dir(module):
-    if i.startswith("_") or not isinstance(getattr(module, i), str):
-      continue
-    output.append(i)
-  return output
-
-
-def check_collisions():
-  defining_modules = defaultdict(list)
-  for module in _MODEL_TAG_MODULES:
-    name = module.__name__
-    for tag in extract_tags(module):
-      defining_modules[tag].append(name)
-
-  duplicate_defs = {k: v for k, v in defining_modules.items() if len(v) > 1}
-  for key in duplicate_defs.keys():
-    print("Variable {} defined multiple times".format(key))
-
-
-def check_format():
-  for tag in sorted(tags.ALL_USED_TAGS):
-    if not TAG_PATTERN.match(tag):
-      print("Malformed tag: {}".format(tag))
-
-
-if __name__ == "__main__":
-  check_collisions()
-  check_format()
-
diff --git a/compliance/verify_submission/mlperf_submission_helper/checks_test.py b/compliance/verify_submission/mlperf_submission_helper/checks_test.py
deleted file mode 100644
index 09f28580d..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/checks_test.py
+++ /dev/null
@@ -1,64 +0,0 @@
-"""Tests checks module."""
-from __future__ import print_function
-
-import os
-import unittest
-
-import checks
-
-
-class TestChecks(unittest.TestCase):
-
-  def test_verify_and_extract_time_not_success(self):
-    """Tests extract the cpu model name."""
-    smi_test = 'unittest_files/10_mixed_results/result_7.txt'
-    smi_test = os.path.join(
-        os.path.dirname(os.path.abspath(__file__)), smi_test)
-    sub_check = checks.SubmissionChecks()
-    dt, start_time = sub_check.verify_and_extract_time(smi_test, 'closed',
-                                                       'ncf')
-    self.assertEqual(dt, checks.INFINITE_TIME)
-    self.assertEqual(start_time, 1541638706.6702664)
-
-  def test_verify_and_extract_time_success(self):
-    """Tests extract the cpu model name."""
-    smi_test = 'unittest_files/10_mixed_results/result_2.txt'
-    smi_test = os.path.join(
-        os.path.dirname(os.path.abspath(__file__)), smi_test)
-    sub_check = checks.SubmissionChecks()
-    dt, start_time = sub_check.verify_and_extract_time(smi_test, 'closed',
-                                                       'ncf')
-    self.assertEqual(dt, 210.3895456790924)
-    self.assertEqual(start_time, 1541635651.95072)
-
-  def test_add_result(self):
-    """Tests adding result to metadata dict."""
-    sub_check = checks.SubmissionChecks()
-    meta = {}
-    meta['entry_name'] = {}
-    meta['entry_name']['result_name'] = {}
-    sub_check._add_result(meta['entry_name']['result_name'],
-                          1,
-                          10.11,
-                          343434343.91)
-    result = meta['entry_name']['result_name'][1]
-    self.assertEqual(result['dt'], 10.11)
-    self.assertEqual(result['start_time'], 343434343.91)
-
-  def test_sort_results(self):
-    """tests sorting results."""
-    results_dict = []
-    results_dict.append(self._create_result_dict(23, 1.993))
-    results_dict.append(self._create_result_dict(55, 19.993))
-    results_dict.append(self._create_result_dict(1, 0.993))
-    results_dict.append(self._create_result_dict(99, 999991.993))
-
-    sub_check = checks.SubmissionChecks()
-    sorted_results = sub_check._sorted_results(results_dict)
-    self.assertEqual(sorted_results, [1, 23, 55, 99])
-
-  def _create_result_dict(self, dt, start_time):
-    result = {}
-    result['dt'] = dt
-    result['start_time'] = start_time
-    return result
diff --git a/compliance/verify_submission/mlperf_submission_helper/test_crypto.py b/compliance/verify_submission/mlperf_submission_helper/test_crypto.py
deleted file mode 100644
index e7f2d2529..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/test_crypto.py
+++ /dev/null
@@ -1,23 +0,0 @@
-import os
-
-import crypto
-
-
-def main():
-    pub_key = "../examples/id_rsa_1024.pub"
-    priv_key = "../examples/id_rsa_1024"
-    raw_file = "../examples/example_log.txt"
-    encrypted_file = "../examples/example_encrypted_file.txt"
-    decrypted_file = "../examples/example_decrypted_file.txt"
-    raw_subm = "../../example_submission"
-    encrypted_subm = "../examples/encrypted_subm"
-    decrypted_subm = "../examples/decrypted_subm"
-
-    # crypto.encrypt_file(pub_key, raw_file, encrypted_file)
-    # crypto.decrypt_file(priv_key, encrypted_file, decrypted_file)
-    crypto.encrypt_submission(pub_key, raw_subm, encrypted_subm)
-    crypto.decrypt_submission(priv_key, encrypted_subm, decrypted_subm)
-
-
-if __name__ == "__main__":
-    main()
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_0.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_0.txt
deleted file mode 100644
index e9b9fd1f8..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_0.txt
+++ /dev/null
@@ -1,157 +0,0 @@
-:::MLPv0.5.0 ncf 1541629769.806975603 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541629798.384909630 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541629798.385934353 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541629827.547986507 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541629829.221915245 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541629829.383048534 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541629829.384613514 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541629829.548022985 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541629829.549335003 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541629829.550442696 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541629829.551510334 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541629829.552565098 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541629829.553709745 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541629830.350249052 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541629830.351327419 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541629830.352361441 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541629834.980409622 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541629834.981530905 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541629841.341938496 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541629852.995471954 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541629853.265025616 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541629853.266101837 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541629853.267829657 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541629854.362714529 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541629870.623628139 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541629874.486646891 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541629892.773094177 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541629892.774147749 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541629892.774910212 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541629902.770447969 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541629902.771357059 (ncf_main.py:298) eval_accuracy : {"value": 0.5306549668312073, "epoch": 0}
-:::MLPv0.5.0 ncf 1541629902.772117376 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541629902.772958279 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541629902.773762226 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541629902.775378704 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541629904.237593412 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541629916.553935766 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541629916.800046206 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541629916.800958633 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541629916.803359985 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541629917.970843792 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541629917.972341061 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541629917.973589420 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541629928.851382732 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541629929.291999578 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541629937.832764149 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541629937.834012508 (ncf_main.py:298) eval_accuracy : {"value": 0.592715859413147, "epoch": 1}
-:::MLPv0.5.0 ncf 1541629937.835010290 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541629937.836045980 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541629937.837090492 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541629937.838867664 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541629942.990456820 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541629943.248872757 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541629943.249788523 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541629943.251494646 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541629944.328391790 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541629944.329252005 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541629944.330059528 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541629955.852084875 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541629955.932825804 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541629965.015548706 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541629965.016636848 (ncf_main.py:298) eval_accuracy : {"value": 0.613258421421051, "epoch": 2}
-:::MLPv0.5.0 ncf 1541629965.017606497 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541629965.018580198 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541629965.019594193 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541629965.021448135 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541629969.613793850 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541629969.866879225 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541629969.867902756 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541629969.869666576 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541629970.971221685 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541629970.972265720 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541629970.973039150 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541629982.223987341 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541629982.718806505 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541629991.377379656 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541629991.379064083 (ncf_main.py:298) eval_accuracy : {"value": 0.6254900693893433, "epoch": 3}
-:::MLPv0.5.0 ncf 1541629991.380533695 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541629991.382159472 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541629991.383623600 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541629991.387483835 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541629996.918225288 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541629997.195878267 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541629997.196891069 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541629997.198756456 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541629998.366734028 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541629998.367702007 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541629998.368456125 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541630009.790888309 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541630010.342236757 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541630018.726572275 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541630018.727678537 (ncf_main.py:298) eval_accuracy : {"value": 0.6303712129592896, "epoch": 4}
-:::MLPv0.5.0 ncf 1541630018.728816271 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541630018.729892015 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541630018.730926037 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541630018.732652426 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541630024.131653070 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541630024.387812853 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541630024.388681412 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541630024.390431404 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541630025.477633238 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541630025.478599548 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541630025.479360104 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541630036.903758764 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541630037.281498194 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541630046.062421560 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541630046.063915491 (ncf_main.py:298) eval_accuracy : {"value": 0.631526529788971, "epoch": 5}
-:::MLPv0.5.0 ncf 1541630046.065145016 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541630046.066528320 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541630046.067807674 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541630046.070402861 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541630051.634691477 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541630051.901442051 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541630051.902601957 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541630051.904453516 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541630053.098103285 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541630053.099166870 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541630053.100075006 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541630064.520211697 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541630065.385507107 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541630073.588383913 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 6}
-:::MLPv0.5.0 ncf 1541630073.589530706 (ncf_main.py:298) eval_accuracy : {"value": 0.6349851489067078, "epoch": 6}
-:::MLPv0.5.0 ncf 1541630073.590510607 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 6}
-:::MLPv0.5.0 ncf 1541630073.591584444 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541630073.592599154 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541630073.594210386 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541630079.577353716 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541630079.842511654 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541630079.843434334 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541630079.845217705 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541630080.996715307 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541630080.997721434 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541630080.998418093 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541630092.849597931 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541630093.108148336 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541630102.393460751 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 7}
-:::MLPv0.5.0 ncf 1541630102.394617081 (ncf_main.py:298) eval_accuracy : {"value": 0.6316348314285278, "epoch": 7}
-:::MLPv0.5.0 ncf 1541630102.395697832 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 7}
-:::MLPv0.5.0 ncf 1541630102.396800518 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541630102.397821426 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541630102.399500370 (ncf_main.py:250) train_epoch : 8
-:::MLPv0.5.0 ncf 1541630106.695773363 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541630106.956234217 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541630106.957199335 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541630106.958822250 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541630108.131520271 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541630108.132481813 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541630108.133234024 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541630119.643209457 (ncf_main.py:288) eval_start : 8
-:::MLPv0.5.0 ncf 1541630120.149136782 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541630128.738948345 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 8}
-:::MLPv0.5.0 ncf 1541630128.740088463 (ncf_main.py:298) eval_accuracy : {"value": 0.6369997262954712, "epoch": 8}
-:::MLPv0.5.0 ncf 1541630128.741153717 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 8}
-:::MLPv0.5.0 ncf 1541630128.742220640 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541630128.743262768 (ncf_main.py:307) eval_stop : 8
-:::MLPv0.5.0 ncf 1541630128.745056629 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541630133.904508591 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_1.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_1.txt
deleted file mode 100644
index 49a39c057..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_1.txt
+++ /dev/null
@@ -1,142 +0,0 @@
-:::MLPv0.5.0 ncf 1541638895.031234026 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541638923.236273527 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541638923.237526655 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541638952.327059031 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541638953.899754524 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541638954.070998192 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541638954.072480679 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541638954.227394104 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541638954.228630543 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541638954.229835272 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541638954.230996609 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541638954.232165098 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541638954.233321428 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541638955.027998924 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638955.028861761 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638955.029631615 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638958.984930515 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541638958.986066341 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541638963.482089281 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638972.804437160 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638973.017053843 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638973.017826796 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638973.019382238 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638973.917438269 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541638989.355852604 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541638993.658016682 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541639011.406038761 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639011.406930923 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639011.407557964 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639020.111869097 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639021.467950821 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541639021.469745398 (ncf_main.py:298) eval_accuracy : {"value": 0.5373484492301941, "epoch": 0}
-:::MLPv0.5.0 ncf 1541639021.471030474 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541639021.472223282 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639021.473330021 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541639021.475375891 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541639032.515873671 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639032.743662119 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639032.744587421 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639032.746376038 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639033.716607571 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639033.717540264 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639033.718369246 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639042.926178694 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639045.802759647 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541639054.909656763 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541639054.910654068 (ncf_main.py:298) eval_accuracy : {"value": 0.5931130051612854, "epoch": 1}
-:::MLPv0.5.0 ncf 1541639054.911531687 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541639054.912374496 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639054.913208246 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541639054.916062355 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541639057.041380882 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639057.275251865 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639057.276146173 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639057.277952194 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639058.295663357 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639058.296445847 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639058.297232866 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639067.436517954 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639070.333390951 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541639079.332252979 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541639079.333365440 (ncf_main.py:298) eval_accuracy : {"value": 0.6087672114372253, "epoch": 2}
-:::MLPv0.5.0 ncf 1541639079.334429979 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541639079.335534573 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639079.336544275 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541639079.338335037 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541639081.162839174 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639081.385721207 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639081.386645555 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639081.388340235 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639082.345177174 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639082.346069336 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639082.346751928 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639091.358014584 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639093.657176971 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541639102.590484858 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541639102.591530800 (ncf_main.py:298) eval_accuracy : {"value": 0.6220602989196777, "epoch": 3}
-:::MLPv0.5.0 ncf 1541639102.592370749 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541639102.593195677 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639102.594020605 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541639102.595510721 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541639104.920227289 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639105.141540527 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639105.142385721 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639105.144113541 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639106.082429647 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639106.083290815 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639106.084046125 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639114.932725191 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639117.956361771 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541639127.011013269 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541639127.012416601 (ncf_main.py:298) eval_accuracy : {"value": 0.6337793469429016, "epoch": 4}
-:::MLPv0.5.0 ncf 1541639127.013852596 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541639127.015208721 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639127.016513586 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541639127.018818855 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541639128.249584675 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639128.469122171 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639128.469949484 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639128.471689939 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639129.393888474 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639129.394659042 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639129.395437241 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639138.089647770 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639141.253282070 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541639150.173605204 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541639150.175257921 (ncf_main.py:298) eval_accuracy : {"value": 0.6301618218421936, "epoch": 5}
-:::MLPv0.5.0 ncf 1541639150.176603079 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541639150.178146601 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639150.179577351 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541639150.182994366 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541639151.672637224 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639151.891428947 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639151.892314196 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639151.894017458 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639152.842330933 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639152.843183756 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639152.844506264 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639161.799849510 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639164.521182537 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541639173.483475447 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 6}
-:::MLPv0.5.0 ncf 1541639173.484593630 (ncf_main.py:298) eval_accuracy : {"value": 0.6319236159324646, "epoch": 6}
-:::MLPv0.5.0 ncf 1541639173.485693693 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 6}
-:::MLPv0.5.0 ncf 1541639173.486776829 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639173.487823009 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541639173.489987850 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541639175.265949488 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639175.495955944 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639175.496826887 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639175.498592615 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639176.444577217 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639176.445418835 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639176.446123600 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639185.473111629 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639187.726145983 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541639196.610095263 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 7}
-:::MLPv0.5.0 ncf 1541639196.611189604 (ncf_main.py:298) eval_accuracy : {"value": 0.6351801156997681, "epoch": 7}
-:::MLPv0.5.0 ncf 1541639196.612210035 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 7}
-:::MLPv0.5.0 ncf 1541639196.613304853 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639196.614780188 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541639196.618208647 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541639202.765043259 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_2.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_2.txt
deleted file mode 100644
index 2e5cc0033..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_2.txt
+++ /dev/null
@@ -1,127 +0,0 @@
-:::MLPv0.5.0 ncf 1541635592.735490084 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541635621.643852949 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541635621.644915342 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541635650.525794268 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541635651.950720072 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541635652.146502495 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541635652.147968292 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541635652.294246912 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541635652.295337915 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541635652.296380043 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541635652.297379255 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541635652.298405170 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541635652.299393654 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541635653.289041758 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635653.290127754 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635653.291014433 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635656.965124607 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541635656.966187954 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541635661.936030626 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635670.668617964 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635670.874290466 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635670.875072479 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635670.876661062 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635671.744560242 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541635686.960785389 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541635690.082700014 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541635707.011981010 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635707.012933254 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635707.013725281 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635715.568417549 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635716.953304768 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541635716.955264330 (ncf_main.py:298) eval_accuracy : {"value": 0.5354422330856323, "epoch": 0}
-:::MLPv0.5.0 ncf 1541635716.956701756 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541635716.958046198 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635716.959423304 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541635716.962947369 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541635726.944634438 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635727.147002935 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635727.147855759 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635727.149498940 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635728.041645765 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635728.042531729 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635728.043314695 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635736.657572508 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635739.158473492 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541635748.078444958 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541635748.079842806 (ncf_main.py:298) eval_accuracy : {"value": 0.5965427756309509, "epoch": 1}
-:::MLPv0.5.0 ncf 1541635748.081184626 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541635748.082456589 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635748.083766937 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541635748.087042809 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541635749.299320221 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635749.496930838 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635749.497620583 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635749.499307632 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635750.400596380 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635750.401340723 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635750.402051210 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635759.082277775 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635762.224506140 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541635771.070236206 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541635771.071095705 (ncf_main.py:298) eval_accuracy : {"value": 0.6109262108802795, "epoch": 2}
-:::MLPv0.5.0 ncf 1541635771.071941614 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541635771.072973728 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635771.073931217 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541635771.075494289 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541635771.735477686 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635771.934694290 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635771.935521603 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635771.937023878 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635772.842210293 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635772.842930555 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635772.844555855 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635781.700635433 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635784.204189301 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541635793.030010939 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541635793.031024933 (ncf_main.py:298) eval_accuracy : {"value": 0.6213093996047974, "epoch": 3}
-:::MLPv0.5.0 ncf 1541635793.031959772 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541635793.032881021 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635793.033894062 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541635793.035282850 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541635794.247166157 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635794.443239450 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635794.444090366 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635794.446002722 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635795.325319529 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635795.326089144 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635795.326739788 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635803.975391626 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635807.174386263 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541635816.100397587 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541635816.101409197 (ncf_main.py:298) eval_accuracy : {"value": 0.628631055355072, "epoch": 4}
-:::MLPv0.5.0 ncf 1541635816.102323532 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541635816.103255987 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635816.104213476 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541635816.105989695 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541635817.220807791 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635817.420272112 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635817.421089411 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635817.422892332 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635818.317587852 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635818.318344355 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635818.318983078 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635827.135539055 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635830.379570484 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541635839.398041487 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541635839.399200678 (ncf_main.py:298) eval_accuracy : {"value": 0.6302556991577148, "epoch": 5}
-:::MLPv0.5.0 ncf 1541635839.400227547 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541635839.401436090 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635839.402719975 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541635839.404899597 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541635841.259696484 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635841.458028316 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635841.458848000 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635841.460329533 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635842.350486755 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635842.351144552 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635842.351816177 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635850.800582409 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635853.518856049 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541635862.334436417 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 6}
-:::MLPv0.5.0 ncf 1541635862.335587025 (ncf_main.py:298) eval_accuracy : {"value": 0.6417508721351624, "epoch": 6}
-:::MLPv0.5.0 ncf 1541635862.336535692 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 6}
-:::MLPv0.5.0 ncf 1541635862.337635994 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635862.338615417 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541635862.340265751 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541635867.477297068 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_3.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_3.txt
deleted file mode 100644
index 67168d7a7..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_3.txt
+++ /dev/null
@@ -1,127 +0,0 @@
-:::MLPv0.5.0 ncf 1541638530.669874907 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541638561.582758665 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541638561.583793640 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541638591.467878819 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541638592.953894854 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541638593.117671490 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541638593.118999958 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541638593.267966270 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541638593.269155025 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541638593.270237923 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541638593.271331310 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541638593.272338390 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541638593.273343563 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541638594.289401293 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638594.290200949 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638594.290861845 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638597.900661469 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541638597.901683331 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541638602.874284029 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638612.318328619 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638612.545264959 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638612.546017408 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638612.547684908 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638613.459522009 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541638629.239745617 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541638633.120075226 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541638650.751036882 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638650.751998901 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638650.752789736 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638660.243846655 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638661.108375072 (ncf_main.py:296) eval_target : {"epoch": 0, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638661.110687017 (ncf_main.py:298) eval_accuracy : {"epoch": 0, "value": 0.5356444120407104}
-:::MLPv0.5.0 ncf 1541638661.112533808 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 0, "value": 999}
-:::MLPv0.5.0 ncf 1541638661.114323378 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638661.116151333 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541638661.120820045 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541638671.748719931 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638671.960242510 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638671.961034775 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638671.962643623 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638672.878739595 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638672.879523277 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638672.880466938 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638682.686112642 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638684.642664671 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541638694.102167130 (ncf_main.py:296) eval_target : {"epoch": 1, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638694.103220463 (ncf_main.py:298) eval_accuracy : {"epoch": 1, "value": 0.58757483959198}
-:::MLPv0.5.0 ncf 1541638694.104175806 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 1, "value": 999}
-:::MLPv0.5.0 ncf 1541638694.105296612 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638694.106267929 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541638694.108056545 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541638696.770808697 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638696.993099689 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638696.994022369 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638696.995660305 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638697.932561159 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638697.933469534 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638697.934192419 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638707.198048353 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638709.624096394 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541638718.868647099 (ncf_main.py:296) eval_target : {"epoch": 2, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638718.869473219 (ncf_main.py:298) eval_accuracy : {"epoch": 2, "value": 0.6153596043586731}
-:::MLPv0.5.0 ncf 1541638718.870290279 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 2, "value": 999}
-:::MLPv0.5.0 ncf 1541638718.871115923 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638718.871890306 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541638718.873318911 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541638720.698725700 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638720.934088707 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638720.934892416 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638720.936585903 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638721.895664454 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638721.896656513 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638721.897491693 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638731.439008951 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638733.294950485 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541638742.417784929 (ncf_main.py:296) eval_target : {"epoch": 3, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638742.418863773 (ncf_main.py:298) eval_accuracy : {"epoch": 3, "value": 0.628912627696991}
-:::MLPv0.5.0 ncf 1541638742.419747114 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 3, "value": 999}
-:::MLPv0.5.0 ncf 1541638742.420757294 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638742.421626806 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541638742.423120737 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541638744.673100710 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638744.884420156 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638744.885207415 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638744.886749744 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638745.789624929 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638745.790396929 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638745.791196823 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638754.539904833 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638757.667276859 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541638766.667538404 (ncf_main.py:296) eval_target : {"epoch": 4, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638766.668629646 (ncf_main.py:298) eval_accuracy : {"epoch": 4, "value": 0.621605396270752}
-:::MLPv0.5.0 ncf 1541638766.669639111 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 4, "value": 999}
-:::MLPv0.5.0 ncf 1541638766.670649767 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638766.671548367 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541638766.673167944 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541638768.128195047 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638768.342603207 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638768.343390465 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638768.345059395 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638769.266900301 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638769.267585278 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638769.268135786 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638778.495992899 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638781.054627657 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541638789.908996582 (ncf_main.py:296) eval_target : {"epoch": 5, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638789.910017729 (ncf_main.py:298) eval_accuracy : {"epoch": 5, "value": 0.6323857307434082}
-:::MLPv0.5.0 ncf 1541638789.910892725 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 5, "value": 999}
-:::MLPv0.5.0 ncf 1541638789.911787748 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638789.912686586 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541638789.914197445 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541638791.479794741 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638791.693878889 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638791.694731474 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638791.696250677 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638792.597578764 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638792.598493814 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638792.599245071 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638801.734756708 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638804.306957722 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541638813.269903660 (ncf_main.py:296) eval_target : {"epoch": 6, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638813.270980597 (ncf_main.py:298) eval_accuracy : {"epoch": 6, "value": 0.6393969655036926}
-:::MLPv0.5.0 ncf 1541638813.271996021 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 6, "value": 999}
-:::MLPv0.5.0 ncf 1541638813.272984028 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638813.273951769 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541638813.275540352 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541638818.417414188 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_4.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_4.txt
deleted file mode 100644
index 8798db362..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_4.txt
+++ /dev/null
@@ -1,187 +0,0 @@
-:::MLPv0.5.0 ncf 1541638735.238862753 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541638763.284983397 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541638763.285973310 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541638792.218703747 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541638793.809119463 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541638793.960701704 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541638793.961995840 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541638794.123754501 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541638794.125000238 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541638794.126100302 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541638794.127219200 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541638794.128327608 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541638794.129439116 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541638795.002657652 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638795.003699064 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638795.004737139 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638799.073868990 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541638799.075083733 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541638804.278205872 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638813.935642004 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638814.165486813 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638814.166306734 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638814.168062687 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638815.102317095 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541638830.649114609 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541638835.380751610 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541638853.196332455 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638853.197302580 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638853.198096514 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638863.102175474 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638863.774731636 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541638863.776791573 (ncf_main.py:298) eval_accuracy : {"value": 0.5355505347251892, "epoch": 0}
-:::MLPv0.5.0 ncf 1541638863.778720856 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541638863.780708790 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638863.782530785 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541638863.787337780 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541638875.236955166 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638875.461423874 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638875.462186337 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638875.464098930 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638876.439197302 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638876.440101624 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638876.440862179 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638886.599519014 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638888.022569418 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541638897.121084213 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541638897.122159481 (ncf_main.py:298) eval_accuracy : {"value": 0.5930985808372498, "epoch": 1}
-:::MLPv0.5.0 ncf 1541638897.123104095 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541638897.124136209 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638897.125144005 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541638897.126816750 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541638900.439061642 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638900.665209532 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638900.666141033 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638900.667827845 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638901.654235363 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638901.655050516 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638901.655836582 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638911.832676649 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638913.336969376 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541638922.436698914 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541638922.437777519 (ncf_main.py:298) eval_accuracy : {"value": 0.6151068806648254, "epoch": 2}
-:::MLPv0.5.0 ncf 1541638922.438806534 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541638922.439902306 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638922.441020012 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541638922.442710400 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541638926.083081245 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638926.307165384 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638926.308060884 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638926.309741020 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638927.284732819 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638927.285611629 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638927.286301136 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638937.344093561 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638938.569292784 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541638947.616512299 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541638947.617608309 (ncf_main.py:298) eval_accuracy : {"value": 0.6168181896209717, "epoch": 3}
-:::MLPv0.5.0 ncf 1541638947.618635178 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541638947.619671345 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638947.620631456 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541638947.622097254 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541638951.270486355 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638951.499932289 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638951.500874281 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638951.502548218 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638952.462307453 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638952.463004112 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638952.463751554 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638962.605659246 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638963.907874823 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541638972.984329224 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541638972.985457182 (ncf_main.py:298) eval_accuracy : {"value": 0.6302845478057861, "epoch": 4}
-:::MLPv0.5.0 ncf 1541638972.986984253 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541638972.988606453 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638972.989959002 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541638972.992711782 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541638976.508027554 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638976.734097958 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638976.734944820 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638976.736789942 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638977.706576109 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638977.707465649 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638977.708255768 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638987.777219772 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638989.068754673 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541638998.090879202 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541638998.092196226 (ncf_main.py:298) eval_accuracy : {"value": 0.6349635124206543, "epoch": 5}
-:::MLPv0.5.0 ncf 1541638998.093468666 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541638998.094665766 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638998.095893860 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541638998.098478794 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541639001.637465715 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639001.867513657 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639001.868403912 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639001.870070696 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639002.827457190 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639002.828288317 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639002.828994989 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639013.103656054 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639014.309495449 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541639023.325899601 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 6}
-:::MLPv0.5.0 ncf 1541639023.327287436 (ncf_main.py:298) eval_accuracy : {"value": 0.6338948607444763, "epoch": 6}
-:::MLPv0.5.0 ncf 1541639023.328736544 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 6}
-:::MLPv0.5.0 ncf 1541639023.330213070 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639023.331642628 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541639023.334031820 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541639026.669128418 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639026.900388718 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639026.901266575 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639026.903734922 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639027.867920399 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639027.868843079 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639027.869573832 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639038.053215265 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639039.575567961 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541639048.613936663 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 7}
-:::MLPv0.5.0 ncf 1541639048.615044355 (ncf_main.py:298) eval_accuracy : {"value": 0.6348768472671509, "epoch": 7}
-:::MLPv0.5.0 ncf 1541639048.616128683 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 7}
-:::MLPv0.5.0 ncf 1541639048.617212534 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639048.618234873 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541639048.620325327 (ncf_main.py:250) train_epoch : 8
-:::MLPv0.5.0 ncf 1541639051.946373701 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639052.170110941 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639052.170968771 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639052.172660589 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639053.131777763 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639053.132664204 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639053.133434534 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639063.435886145 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639064.853965521 (ncf_main.py:288) eval_start : 8
-:::MLPv0.5.0 ncf 1541639073.796424389 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 8}
-:::MLPv0.5.0 ncf 1541639073.797993422 (ncf_main.py:298) eval_accuracy : {"value": 0.6314976215362549, "epoch": 8}
-:::MLPv0.5.0 ncf 1541639073.799522161 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 8}
-:::MLPv0.5.0 ncf 1541639073.801043034 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639073.802507401 (ncf_main.py:307) eval_stop : 8
-:::MLPv0.5.0 ncf 1541639073.806848288 (ncf_main.py:250) train_epoch : 9
-:::MLPv0.5.0 ncf 1541639077.021913528 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639077.248788118 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639077.249706030 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639077.251465797 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639078.222069263 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639078.222922325 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639078.223669529 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639088.445314884 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639089.939385414 (ncf_main.py:288) eval_start : 9
-:::MLPv0.5.0 ncf 1541639098.979931116 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 9}
-:::MLPv0.5.0 ncf 1541639098.980998755 (ncf_main.py:298) eval_accuracy : {"value": 0.6349129676818848, "epoch": 9}
-:::MLPv0.5.0 ncf 1541639098.981842041 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 9}
-:::MLPv0.5.0 ncf 1541639098.982664585 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639098.983457565 (ncf_main.py:307) eval_stop : 9
-:::MLPv0.5.0 ncf 1541639098.985089540 (ncf_main.py:250) train_epoch : 10
-:::MLPv0.5.0 ncf 1541639102.237427950 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639102.472718000 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639102.473574400 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639102.475203753 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639103.493594646 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639103.494433880 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639103.495185614 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639113.654072762 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639115.319412231 (ncf_main.py:288) eval_start : 10
-:::MLPv0.5.0 ncf 1541639124.446080923 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 10}
-:::MLPv0.5.0 ncf 1541639124.446999550 (ncf_main.py:298) eval_accuracy : {"value": 0.6357361078262329, "epoch": 10}
-:::MLPv0.5.0 ncf 1541639124.447862625 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 10}
-:::MLPv0.5.0 ncf 1541639124.449436665 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639124.450917006 (ncf_main.py:307) eval_stop : 10
-:::MLPv0.5.0 ncf 1541639124.454472303 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541639129.607702017 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_5.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_5.txt
deleted file mode 100644
index 18325d64f..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_5.txt
+++ /dev/null
@@ -1,157 +0,0 @@
-:::MLPv0.5.0 ncf 1541638763.440427065 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541638790.779615164 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541638790.780507803 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541638819.940380812 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541638821.379003525 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541638821.534843683 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541638821.536063910 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541638821.679996252 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541638821.681133509 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541638821.682166576 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541638821.683179617 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541638821.684192419 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541638821.685162306 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541638822.613849878 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638822.614837170 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638822.615612984 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638826.204838276 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541638826.205809355 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541638831.776727200 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638841.261714458 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638841.475725651 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638841.476668119 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638841.478289366 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638842.397707701 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541638858.403058529 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541638861.475663424 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541638878.624043465 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638878.624924421 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638878.625625849 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638888.412789345 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638888.917968035 (ncf_main.py:296) eval_target : {"epoch": 0, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638888.920070887 (ncf_main.py:298) eval_accuracy : {"epoch": 0, "value": 0.5279328227043152}
-:::MLPv0.5.0 ncf 1541638888.921729088 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 0, "value": 999}
-:::MLPv0.5.0 ncf 1541638888.923309326 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638888.924749374 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541638888.929114819 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541638900.195536137 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638900.399672270 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638900.400523186 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638900.402125120 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638901.344372511 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638901.345240355 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638901.345992088 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638911.365325689 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638913.131508350 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541638922.109789371 (ncf_main.py:296) eval_target : {"epoch": 1, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638922.110926628 (ncf_main.py:298) eval_accuracy : {"epoch": 1, "value": 0.5782313942909241}
-:::MLPv0.5.0 ncf 1541638922.111940145 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 1, "value": 999}
-:::MLPv0.5.0 ncf 1541638922.112982035 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638922.113954067 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541638922.115504980 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541638924.538333178 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638924.745608091 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638924.746380806 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638924.749269247 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638925.686165333 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638925.686889172 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638925.687494993 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638935.590030909 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638937.294821262 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541638946.289694071 (ncf_main.py:296) eval_target : {"epoch": 2, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638946.290822268 (ncf_main.py:298) eval_accuracy : {"epoch": 2, "value": 0.6097131371498108}
-:::MLPv0.5.0 ncf 1541638946.291926384 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 2, "value": 999}
-:::MLPv0.5.0 ncf 1541638946.293175459 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638946.294192553 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541638946.296341181 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541638948.400600910 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638948.611230612 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638948.612046242 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638948.613714218 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638949.553132296 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638949.553850174 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638949.554521561 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638959.460954189 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638961.404874086 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541638970.316626787 (ncf_main.py:296) eval_target : {"epoch": 3, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638970.318053722 (ncf_main.py:298) eval_accuracy : {"epoch": 3, "value": 0.6186810731887817}
-:::MLPv0.5.0 ncf 1541638970.319359541 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 3, "value": 999}
-:::MLPv0.5.0 ncf 1541638970.320714712 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638970.322079659 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541638970.325397015 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541638972.837314129 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638973.050715685 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638973.051517487 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638973.053102970 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638974.001861095 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638974.002641201 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638974.003303289 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638983.682631969 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638985.509771109 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541638994.446334600 (ncf_main.py:296) eval_target : {"epoch": 4, "value": 0.635}
-:::MLPv0.5.0 ncf 1541638994.447399855 (ncf_main.py:298) eval_accuracy : {"epoch": 4, "value": 0.6311221718788147}
-:::MLPv0.5.0 ncf 1541638994.448939085 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 4, "value": 999}
-:::MLPv0.5.0 ncf 1541638994.450215578 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638994.451440334 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541638994.454709053 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541638996.432271004 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638996.635140657 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638996.635883808 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638996.637431145 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638997.594397783 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638997.595444202 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638997.596254110 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639007.279758215 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639008.539303541 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541639017.558956861 (ncf_main.py:296) eval_target : {"epoch": 5, "value": 0.635}
-:::MLPv0.5.0 ncf 1541639017.560356617 (ncf_main.py:298) eval_accuracy : {"epoch": 5, "value": 0.6311654448509216}
-:::MLPv0.5.0 ncf 1541639017.561654568 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 5, "value": 999}
-:::MLPv0.5.0 ncf 1541639017.562897205 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639017.564193726 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541639017.567703962 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541639020.146848440 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639020.354084730 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639020.354932070 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639020.356535912 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639021.275757551 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639021.276599407 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639021.277253628 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639031.074332237 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639032.801159382 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541639041.756096125 (ncf_main.py:296) eval_target : {"epoch": 6, "value": 0.635}
-:::MLPv0.5.0 ncf 1541639041.756878853 (ncf_main.py:298) eval_accuracy : {"epoch": 6, "value": 0.6286599040031433}
-:::MLPv0.5.0 ncf 1541639041.757618189 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 6, "value": 999}
-:::MLPv0.5.0 ncf 1541639041.758368731 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639041.759097338 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541639041.760601759 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541639043.498859882 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639043.696454525 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639043.697260141 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639043.698834181 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639044.589266777 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639044.590030670 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639044.590780020 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639054.153103590 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639055.968831062 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541639064.915617704 (ncf_main.py:296) eval_target : {"epoch": 7, "value": 0.635}
-:::MLPv0.5.0 ncf 1541639064.916630030 (ncf_main.py:298) eval_accuracy : {"epoch": 7, "value": 0.6326096057891846}
-:::MLPv0.5.0 ncf 1541639064.917640924 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 7, "value": 999}
-:::MLPv0.5.0 ncf 1541639064.918621540 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639064.919554234 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541639064.921204090 (ncf_main.py:250) train_epoch : 8
-:::MLPv0.5.0 ncf 1541639067.292711258 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639067.498054266 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639067.498902559 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639067.500632524 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639068.435379267 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639068.436191559 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639068.436933756 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639078.104113579 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639080.104954004 (ncf_main.py:288) eval_start : 8
-:::MLPv0.5.0 ncf 1541639089.076507092 (ncf_main.py:296) eval_target : {"epoch": 8, "value": 0.635}
-:::MLPv0.5.0 ncf 1541639089.077627182 (ncf_main.py:298) eval_accuracy : {"epoch": 8, "value": 0.6405161023139954}
-:::MLPv0.5.0 ncf 1541639089.078667402 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 8, "value": 999}
-:::MLPv0.5.0 ncf 1541639089.079668045 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639089.080657959 (ncf_main.py:307) eval_stop : 8
-:::MLPv0.5.0 ncf 1541639089.082299232 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541639095.228493929 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_6.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_6.txt
deleted file mode 100644
index 45f4be1a5..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_6.txt
+++ /dev/null
@@ -1,112 +0,0 @@
-:::MLPv0.5.0 ncf 1541635104.310555458 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541635131.471904993 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541635131.472935915 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541635159.243577003 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541635160.774804592 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541635160.940083504 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541635160.941361666 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541635161.090290546 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541635161.091511726 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541635161.092599154 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541635161.093669176 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541635161.094815493 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541635161.095993996 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541635162.026654720 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635162.027592182 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635162.028356314 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635165.954928398 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541635165.956025600 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541635171.832559586 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635182.169102907 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635182.401242733 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635182.402066946 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635182.403608084 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635183.356632233 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541635199.252304316 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541635202.663941860 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541635220.301067829 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635220.302071810 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635220.303174496 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635230.153288841 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541635230.154069424 (ncf_main.py:298) eval_accuracy : {"value": 0.5405616164207458, "epoch": 0}
-:::MLPv0.5.0 ncf 1541635230.154792309 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541635230.155698061 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635230.156645060 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541635230.158128262 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541635230.840999365 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635242.693369865 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635242.926271439 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635242.927160025 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635242.928939104 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635243.977257490 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635243.977997303 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635243.978714228 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635254.760675192 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635255.229307890 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541635264.206905127 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541635264.208005667 (ncf_main.py:298) eval_accuracy : {"value": 0.5901886820793152, "epoch": 1}
-:::MLPv0.5.0 ncf 1541635264.208961487 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541635264.209945202 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635264.210886955 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541635264.212466002 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541635268.100805759 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635268.339647532 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635268.340467215 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635268.341966867 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635269.361189842 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635269.362025023 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635269.362709761 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635279.805070877 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635281.209762573 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541635290.199726343 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541635290.200830936 (ncf_main.py:298) eval_accuracy : {"value": 0.6160889267921448, "epoch": 2}
-:::MLPv0.5.0 ncf 1541635290.201836824 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541635290.202774525 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635290.203707695 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541635290.205309153 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541635293.114334106 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635293.349895716 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635293.350741863 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635293.352239847 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635294.374027967 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635294.374819994 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635294.375455618 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635305.165984392 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635306.245191574 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541635315.263355970 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541635315.264427185 (ncf_main.py:298) eval_accuracy : {"value": 0.6182406544685364, "epoch": 3}
-:::MLPv0.5.0 ncf 1541635315.265385866 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541635315.266289711 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635315.267217875 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541635315.271013260 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541635318.532382250 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635318.769673824 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635318.770560503 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635318.772199631 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635319.784989834 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635319.785841465 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635319.786663532 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635330.223200560 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635331.300731897 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541635340.334658146 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541635340.335757256 (ncf_main.py:298) eval_accuracy : {"value": 0.6218942403793335, "epoch": 4}
-:::MLPv0.5.0 ncf 1541635340.337112665 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541635340.338456869 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635340.339895725 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541635340.343821287 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541635343.544234753 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541635343.779250383 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541635343.780206680 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541635343.781805277 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541635344.787969351 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541635344.788851500 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541635344.789602280 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541635355.629423380 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541635356.461983919 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541635365.410529137 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541635365.411608458 (ncf_main.py:298) eval_accuracy : {"value": 0.6367903351783752, "epoch": 5}
-:::MLPv0.5.0 ncf 1541635365.412614107 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541635365.413582802 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541635365.414509773 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541635365.416010141 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541635371.566162825 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_7.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_7.txt
deleted file mode 100644
index 800d26995..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_7.txt
+++ /dev/null
@@ -1,232 +0,0 @@
-:::MLPv0.5.0 ncf 1541638648.550307274 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541638677.071330070 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541638677.072561979 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541638705.039521694 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541638706.670266390 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541638706.826765537 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541638706.828184128 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541638706.985775471 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541638706.987082481 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541638706.988264561 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541638706.989395857 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541638706.990535021 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541638706.991703749 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541638707.827156305 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638707.828113556 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638707.828854799 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638711.912560463 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541638711.913695574 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541638717.573549032 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638727.377744675 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638727.607499361 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638727.608463526 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638727.610274553 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638728.564833403 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541638744.498449087 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541638749.060900688 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541638768.275154829 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638768.276139736 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638768.276959896 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638777.796409845 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638778.578298330 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541638778.579566717 (ncf_main.py:298) eval_accuracy : {"value": 0.5355721712112427, "epoch": 0}
-:::MLPv0.5.0 ncf 1541638778.580665112 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541638778.581835747 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638778.582897902 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541638778.585196495 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541638789.774667978 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638790.003770113 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638790.004733562 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638790.006499290 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638791.009516478 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638791.010442019 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638791.011214495 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638800.901390076 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638802.952246904 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541638812.051494837 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541638812.052587509 (ncf_main.py:298) eval_accuracy : {"value": 0.5825492739677429, "epoch": 1}
-:::MLPv0.5.0 ncf 1541638812.054286957 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541638812.056334496 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638812.058329582 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541638812.063334703 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541638815.112897158 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638815.351840734 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638815.352747679 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638815.354386568 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638816.401205540 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638816.402038574 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638816.402697563 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638826.358629942 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638828.482375145 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541638837.589383125 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541638837.590895414 (ncf_main.py:298) eval_accuracy : {"value": 0.6043915748596191, "epoch": 2}
-:::MLPv0.5.0 ncf 1541638837.592641830 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541638837.594459057 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638837.596236944 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541638837.600217104 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541638840.670164347 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638840.901393890 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638840.902322769 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638840.904255867 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638841.919704437 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638841.920623541 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638841.921340466 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638851.890675306 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638853.890926838 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541638863.027609110 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541638863.028797626 (ncf_main.py:298) eval_accuracy : {"value": 0.6211938261985779, "epoch": 3}
-:::MLPv0.5.0 ncf 1541638863.029858112 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541638863.030918360 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638863.032007933 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541638863.035703182 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541638865.837581158 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638866.066373348 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638866.067242622 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638866.068979740 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638867.089495420 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638867.090789557 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638867.091881752 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638876.884055376 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638878.318868637 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541638887.453387022 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541638887.454644918 (ncf_main.py:298) eval_accuracy : {"value": 0.6266164779663086, "epoch": 4}
-:::MLPv0.5.0 ncf 1541638887.455842257 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541638887.457065105 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638887.458268166 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541638887.460571527 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541638891.187935352 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638891.417843580 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638891.418778896 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638891.420610905 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638892.425272942 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638892.426149368 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638892.426887274 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638902.111029863 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638903.784252882 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541638913.010018349 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541638913.011149883 (ncf_main.py:298) eval_accuracy : {"value": 0.626602053642273, "epoch": 5}
-:::MLPv0.5.0 ncf 1541638913.012211800 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541638913.013282776 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638913.014289141 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541638913.015935421 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541638916.771363020 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638917.004644871 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638917.005646229 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638917.008382082 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638918.012643337 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638918.013614655 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638918.014599323 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638927.594116926 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638929.446775675 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541638938.570993185 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 6}
-:::MLPv0.5.0 ncf 1541638938.572448015 (ncf_main.py:298) eval_accuracy : {"value": 0.6349635124206543, "epoch": 6}
-:::MLPv0.5.0 ncf 1541638938.573783398 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 6}
-:::MLPv0.5.0 ncf 1541638938.575094700 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638938.576342583 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541638938.578654766 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541638941.692667484 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638941.922924995 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638941.923849344 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638941.925620794 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638942.935578108 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638942.936476469 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638942.937253952 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638952.531521082 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638954.897549868 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541638964.103841782 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 7}
-:::MLPv0.5.0 ncf 1541638964.105089188 (ncf_main.py:298) eval_accuracy : {"value": 0.6335626840591431, "epoch": 7}
-:::MLPv0.5.0 ncf 1541638964.106173038 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 7}
-:::MLPv0.5.0 ncf 1541638964.107293367 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638964.108463049 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541638964.110485792 (ncf_main.py:250) train_epoch : 8
-:::MLPv0.5.0 ncf 1541638966.973208904 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638967.203658581 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638967.204607010 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638967.206352234 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638968.233442068 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638968.234580517 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638968.235488653 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541638978.180021524 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541638979.381479025 (ncf_main.py:288) eval_start : 8
-:::MLPv0.5.0 ncf 1541638988.413835287 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 8}
-:::MLPv0.5.0 ncf 1541638988.415410757 (ncf_main.py:298) eval_accuracy : {"value": 0.6319958567619324, "epoch": 8}
-:::MLPv0.5.0 ncf 1541638988.416940689 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 8}
-:::MLPv0.5.0 ncf 1541638988.418448210 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541638988.419909000 (ncf_main.py:307) eval_stop : 8
-:::MLPv0.5.0 ncf 1541638988.423610687 (ncf_main.py:250) train_epoch : 9
-:::MLPv0.5.0 ncf 1541638992.395078421 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541638992.628517866 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541638992.629472017 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541638992.631278753 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541638993.648834944 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541638993.649730921 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541638993.650494337 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639003.502723694 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639005.726691961 (ncf_main.py:288) eval_start : 9
-:::MLPv0.5.0 ncf 1541639014.860283136 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 9}
-:::MLPv0.5.0 ncf 1541639014.861481667 (ncf_main.py:298) eval_accuracy : {"value": 0.6324362754821777, "epoch": 9}
-:::MLPv0.5.0 ncf 1541639014.862538099 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 9}
-:::MLPv0.5.0 ncf 1541639014.863608122 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639014.864690304 (ncf_main.py:307) eval_stop : 9
-:::MLPv0.5.0 ncf 1541639014.866374016 (ncf_main.py:250) train_epoch : 10
-:::MLPv0.5.0 ncf 1541639017.435810089 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639017.667629004 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639017.668562889 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639017.670313597 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639018.692301035 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639018.693156958 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639018.693868399 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639028.311717987 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639030.179833651 (ncf_main.py:288) eval_start : 10
-:::MLPv0.5.0 ncf 1541639039.233758688 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 10}
-:::MLPv0.5.0 ncf 1541639039.234900475 (ncf_main.py:298) eval_accuracy : {"value": 0.631252110004425, "epoch": 10}
-:::MLPv0.5.0 ncf 1541639039.236017466 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 10}
-:::MLPv0.5.0 ncf 1541639039.237156153 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639039.238312483 (ncf_main.py:307) eval_stop : 10
-:::MLPv0.5.0 ncf 1541639039.239964247 (ncf_main.py:250) train_epoch : 11
-:::MLPv0.5.0 ncf 1541639042.740925789 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639042.973051310 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639042.973954201 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639042.975731373 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639043.988547564 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639043.989431620 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639043.990193844 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639054.026009560 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639055.715245485 (ncf_main.py:288) eval_start : 11
-:::MLPv0.5.0 ncf 1541639064.891018629 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 11}
-:::MLPv0.5.0 ncf 1541639064.892221928 (ncf_main.py:298) eval_accuracy : {"value": 0.6324723958969116, "epoch": 11}
-:::MLPv0.5.0 ncf 1541639064.893285751 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 11}
-:::MLPv0.5.0 ncf 1541639064.894388676 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639064.895438671 (ncf_main.py:307) eval_stop : 11
-:::MLPv0.5.0 ncf 1541639064.897495270 (ncf_main.py:250) train_epoch : 12
-:::MLPv0.5.0 ncf 1541639068.386284590 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639068.620766878 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639068.621762514 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639068.623532295 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639069.652983189 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639069.653871775 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639069.655254602 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639079.759501696 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639081.394753456 (ncf_main.py:288) eval_start : 12
-:::MLPv0.5.0 ncf 1541639090.601745129 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 12}
-:::MLPv0.5.0 ncf 1541639090.602892160 (ncf_main.py:298) eval_accuracy : {"value": 0.6331583261489868, "epoch": 12}
-:::MLPv0.5.0 ncf 1541639090.603966951 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 12}
-:::MLPv0.5.0 ncf 1541639090.605060101 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639090.606139421 (ncf_main.py:307) eval_stop : 12
-:::MLPv0.5.0 ncf 1541639090.607813358 (ncf_main.py:250) train_epoch : 13
-:::MLPv0.5.0 ncf 1541639093.529235601 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541639093.761765242 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541639093.762660503 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541639093.764389277 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541639094.784854174 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541639094.785711288 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541639094.786442280 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541639104.575978756 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541639106.884737015 (ncf_main.py:288) eval_start : 13
-:::MLPv0.5.0 ncf 1541639115.941901922 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 13}
-:::MLPv0.5.0 ncf 1541639115.943205118 (ncf_main.py:298) eval_accuracy : {"value": 0.6290282011032104, "epoch": 13}
-:::MLPv0.5.0 ncf 1541639115.944553137 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 13}
-:::MLPv0.5.0 ncf 1541639115.945715904 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541639115.946871042 (ncf_main.py:307) eval_stop : 13
-:::MLPv0.5.0 ncf 1541639115.948885441 (ncf_main.py:322) run_stop : {"success": false}
-:::MLPv0.5.0 ncf 1541639122.101526499 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_8.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_8.txt
deleted file mode 100644
index 788d5ac51..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_8.txt
+++ /dev/null
@@ -1,187 +0,0 @@
-:::MLPv0.5.0 ncf 1541633784.076412439 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541633812.331842184 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541633812.332926035 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541633840.689930201 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541633842.217522621 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541633842.384579897 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541633842.385924578 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541633842.538933277 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541633842.540210247 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541633842.541359186 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541633842.542474985 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541633842.543566227 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541633842.544742823 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541633843.494700670 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541633843.495620251 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541633843.496549845 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541633847.428571463 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541633847.429612398 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541633852.794543266 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541633862.796217203 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541633863.044460297 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541633863.045340061 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541633863.047086716 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541633864.020861387 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541633880.195640802 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541633884.594573736 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541633903.524662733 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541633903.525588751 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541633903.526331902 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541633912.792108297 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541633913.999493361 (ncf_main.py:296) eval_target : {"epoch": 0, "value": 0.635}
-:::MLPv0.5.0 ncf 1541633914.001715422 (ncf_main.py:298) eval_accuracy : {"epoch": 0, "value": 0.5355144143104553}
-:::MLPv0.5.0 ncf 1541633914.003288031 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 0, "value": 999}
-:::MLPv0.5.0 ncf 1541633914.004815102 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541633914.006219625 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541633914.010214329 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541633925.758931160 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541633926.001628876 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541633926.002696991 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541633926.004520893 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541633927.017638206 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541633927.018500805 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541633927.019226551 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541633936.503736973 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541633938.675363541 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541633947.942579508 (ncf_main.py:296) eval_target : {"epoch": 1, "value": 0.635}
-:::MLPv0.5.0 ncf 1541633947.944736719 (ncf_main.py:298) eval_accuracy : {"epoch": 1, "value": 0.5933079719543457}
-:::MLPv0.5.0 ncf 1541633947.946526766 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 1, "value": 999}
-:::MLPv0.5.0 ncf 1541633947.948253393 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541633947.950294495 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541633947.955521345 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541633951.192464828 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541633951.427693605 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541633951.428581238 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541633951.430352211 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541633952.440587044 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541633952.441425323 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541633952.442149401 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541633962.389586926 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541633964.485746145 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541633973.832741261 (ncf_main.py:296) eval_target : {"epoch": 2, "value": 0.635}
-:::MLPv0.5.0 ncf 1541633973.834342957 (ncf_main.py:298) eval_accuracy : {"epoch": 2, "value": 0.6142404079437256}
-:::MLPv0.5.0 ncf 1541633973.835930586 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 2, "value": 999}
-:::MLPv0.5.0 ncf 1541633973.837467670 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541633973.838932276 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541633973.842877150 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541633976.868446589 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541633977.108947992 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541633977.109932899 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541633977.111764193 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541633978.124878407 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541633978.125756741 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541633978.126549244 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541633987.882139444 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541633990.403650284 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541633999.752817392 (ncf_main.py:296) eval_target : {"epoch": 3, "value": 0.635}
-:::MLPv0.5.0 ncf 1541633999.754131317 (ncf_main.py:298) eval_accuracy : {"epoch": 3, "value": 0.6230350732803345}
-:::MLPv0.5.0 ncf 1541633999.755548477 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 3, "value": 999}
-:::MLPv0.5.0 ncf 1541633999.756847858 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541633999.758169174 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541633999.760797262 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541634001.934432507 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634002.172694921 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634002.173647642 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634002.176377296 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634003.209175110 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634003.210082054 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634003.210810661 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634012.724496841 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634015.359450340 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541634024.623889685 (ncf_main.py:296) eval_target : {"epoch": 4, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634024.625084400 (ncf_main.py:298) eval_accuracy : {"epoch": 4, "value": 0.624255359172821}
-:::MLPv0.5.0 ncf 1541634024.626185179 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 4, "value": 999}
-:::MLPv0.5.0 ncf 1541634024.627274275 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634024.628348112 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541634024.630026102 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541634027.304376364 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634027.554424047 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634027.555421829 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634027.557218790 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634028.599627018 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634028.600513935 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634028.601523399 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634038.247190952 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634040.204885483 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541634049.454578400 (ncf_main.py:296) eval_target : {"epoch": 5, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634049.455724716 (ncf_main.py:298) eval_accuracy : {"epoch": 5, "value": 0.6309777498245239}
-:::MLPv0.5.0 ncf 1541634049.456771612 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 5, "value": 999}
-:::MLPv0.5.0 ncf 1541634049.457925081 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634049.458989143 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541634049.460813522 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541634052.598333120 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634052.838178158 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634052.839139462 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634052.840905190 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634053.873098135 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634053.873992443 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634053.874759674 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634063.560905457 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634066.015788794 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541634075.329430342 (ncf_main.py:296) eval_target : {"epoch": 6, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634075.330686569 (ncf_main.py:298) eval_accuracy : {"epoch": 6, "value": 0.6332811117172241}
-:::MLPv0.5.0 ncf 1541634075.331876755 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 6, "value": 999}
-:::MLPv0.5.0 ncf 1541634075.332991362 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634075.334129810 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541634075.336100101 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541634078.215678692 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634078.458297729 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634078.459167004 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634078.460916281 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634079.510891676 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634079.511835337 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634079.512629986 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634089.159586430 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634091.814576387 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541634100.890487909 (ncf_main.py:296) eval_target : {"epoch": 7, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634100.891679287 (ncf_main.py:298) eval_accuracy : {"epoch": 7, "value": 0.6258655786514282}
-:::MLPv0.5.0 ncf 1541634100.892809868 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 7, "value": 999}
-:::MLPv0.5.0 ncf 1541634100.893930674 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634100.895038366 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541634100.896654367 (ncf_main.py:250) train_epoch : 8
-:::MLPv0.5.0 ncf 1541634103.229527712 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634103.466492653 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634103.467323542 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634103.468997955 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634104.497328281 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634104.498208523 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634104.498906136 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634113.913375139 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634116.370789051 (ncf_main.py:288) eval_start : 8
-:::MLPv0.5.0 ncf 1541634125.495560646 (ncf_main.py:296) eval_target : {"epoch": 8, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634125.496621370 (ncf_main.py:298) eval_accuracy : {"epoch": 8, "value": 0.6348840594291687}
-:::MLPv0.5.0 ncf 1541634125.497645378 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 8, "value": 999}
-:::MLPv0.5.0 ncf 1541634125.498773336 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634125.499867916 (ncf_main.py:307) eval_stop : 8
-:::MLPv0.5.0 ncf 1541634125.501574516 (ncf_main.py:250) train_epoch : 9
-:::MLPv0.5.0 ncf 1541634128.347003698 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634128.588598967 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634128.589591026 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634128.591367006 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634129.643951416 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634129.645126343 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634129.646170139 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634138.920884132 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634140.920762300 (ncf_main.py:288) eval_start : 9
-:::MLPv0.5.0 ncf 1541634150.038336277 (ncf_main.py:296) eval_target : {"epoch": 9, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634150.039472580 (ncf_main.py:298) eval_accuracy : {"epoch": 9, "value": 0.6306816935539246}
-:::MLPv0.5.0 ncf 1541634150.040536165 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 9, "value": 999}
-:::MLPv0.5.0 ncf 1541634150.041595697 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634150.042622089 (ncf_main.py:307) eval_stop : 9
-:::MLPv0.5.0 ncf 1541634150.044364929 (ncf_main.py:250) train_epoch : 10
-:::MLPv0.5.0 ncf 1541634153.084785700 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541634153.321539164 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541634153.322626829 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541634153.324467182 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541634154.330637217 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541634154.331571579 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541634154.332372904 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541634164.011516094 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541634166.685122490 (ncf_main.py:288) eval_start : 10
-:::MLPv0.5.0 ncf 1541634175.769804001 (ncf_main.py:296) eval_target : {"epoch": 10, "value": 0.635}
-:::MLPv0.5.0 ncf 1541634175.771140099 (ncf_main.py:298) eval_accuracy : {"epoch": 10, "value": 0.6350429058074951}
-:::MLPv0.5.0 ncf 1541634175.772264957 (ncf_main.py:301) eval_hp_num_neg : {"epoch": 10, "value": 999}
-:::MLPv0.5.0 ncf 1541634175.773369312 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541634175.774472713 (ncf_main.py:307) eval_stop : 10
-:::MLPv0.5.0 ncf 1541634175.776590347 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541634181.931392193 (ncf_main.py:328) run_final 
diff --git a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_9.txt b/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_9.txt
deleted file mode 100644
index d0a51a810..000000000
--- a/compliance/verify_submission/mlperf_submission_helper/unittest_files/10_mixed_results/result_9.txt
+++ /dev/null
@@ -1,142 +0,0 @@
-:::MLPv0.5.0 ncf 1541631499.260495424 (data_preprocessing.py:139) preproc_hp_min_ratings : 20
-:::MLPv0.5.0 ncf 1541631528.275058508 (data_preprocessing.py:155) preproc_hp_num_eval : 999
-:::MLPv0.5.0 ncf 1541631528.276078463 (data_preprocessing.py:158) preproc_hp_sample_eval_replacement : true
-:::MLPv0.5.0 ncf 1541631556.569270849 (data_preprocessing.py:514) run_clear_caches 
-:::MLPv0.5.0 ncf 1541631558.095288277 (data_preprocessing.py:516) run_start 
-:::MLPv0.5.0 ncf 1541631558.260651350 (neumf_model.py:189) model_hp_mf_dim : 64
-:::MLPv0.5.0 ncf 1541631558.261936665 (neumf_model.py:191) model_hp_mlp_layer_sizes : [256, 256, 128, 64]
-:::MLPv0.5.0 ncf 1541631558.414551020 (neumf_model.py:118) opt_name : "adam"
-:::MLPv0.5.0 ncf 1541631558.415807247 (neumf_model.py:120) opt_learning_rate : 0.00395706
-:::MLPv0.5.0 ncf 1541631558.417001963 (neumf_model.py:122) opt_hp_Adam_beta1 : 0.779661
-:::MLPv0.5.0 ncf 1541631558.418153763 (neumf_model.py:124) opt_hp_Adam_beta2 : 0.895586
-:::MLPv0.5.0 ncf 1541631558.419267416 (neumf_model.py:126) opt_hp_Adam_epsilon : 1.45039e-07
-:::MLPv0.5.0 ncf 1541631558.420404434 (neumf_model.py:135) model_hp_loss_fn : "binary_cross_entropy"
-:::MLPv0.5.0 ncf 1541631559.281426907 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631559.282351732 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631559.283097506 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631563.618130684 (ncf_main.py:243) train_loop 
-:::MLPv0.5.0 ncf 1541631563.619222164 (ncf_main.py:250) train_epoch : 0
-:::MLPv0.5.0 ncf 1541631568.264500618 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631577.735880613 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631577.957475662 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631577.958349943 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631577.960924625 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631578.871345997 (data_async_generation.py:240) input_step_eval_neg_gen 
-:::MLPv0.5.0 ncf 1541631595.015643597 (ncf_main.py:288) eval_start : 0
-:::MLPv0.5.0 ncf 1541631599.068238020 (data_async_generation.py:320) eval_size : 138493000
-:::MLPv0.5.0 ncf 1541631616.733568192 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631616.734480381 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631616.735254288 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631625.623954535 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631627.058417320 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 0}
-:::MLPv0.5.0 ncf 1541631627.060239077 (ncf_main.py:298) eval_accuracy : {"value": 0.5299473404884338, "epoch": 0}
-:::MLPv0.5.0 ncf 1541631627.061510324 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 0}
-:::MLPv0.5.0 ncf 1541631627.062723875 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631627.063873053 (ncf_main.py:307) eval_stop : 0
-:::MLPv0.5.0 ncf 1541631627.066453695 (ncf_main.py:250) train_epoch : 1
-:::MLPv0.5.0 ncf 1541631637.954942942 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631638.170925140 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631638.171757221 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631638.173471928 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631639.173291922 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631639.174464464 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631639.175425291 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631648.537451267 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631650.270410061 (ncf_main.py:288) eval_start : 1
-:::MLPv0.5.0 ncf 1541631659.307844400 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 1}
-:::MLPv0.5.0 ncf 1541631659.308756351 (ncf_main.py:298) eval_accuracy : {"value": 0.5876036882400513, "epoch": 1}
-:::MLPv0.5.0 ncf 1541631659.309592724 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 1}
-:::MLPv0.5.0 ncf 1541631659.310423613 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631659.311274529 (ncf_main.py:307) eval_stop : 1
-:::MLPv0.5.0 ncf 1541631659.312756062 (ncf_main.py:250) train_epoch : 2
-:::MLPv0.5.0 ncf 1541631662.343611479 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631662.560714722 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631662.561597109 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631662.563328505 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631663.508899450 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631663.509758949 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631663.510467052 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631672.696743250 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631675.625155210 (ncf_main.py:288) eval_start : 2
-:::MLPv0.5.0 ncf 1541631684.675855875 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 2}
-:::MLPv0.5.0 ncf 1541631684.677035093 (ncf_main.py:298) eval_accuracy : {"value": 0.617013156414032, "epoch": 2}
-:::MLPv0.5.0 ncf 1541631684.678092957 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 2}
-:::MLPv0.5.0 ncf 1541631684.679159164 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631684.680258751 (ncf_main.py:307) eval_stop : 2
-:::MLPv0.5.0 ncf 1541631684.683620453 (ncf_main.py:250) train_epoch : 3
-:::MLPv0.5.0 ncf 1541631686.192053556 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631686.415297508 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631686.416174650 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631686.417960405 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631687.385180235 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631687.385957003 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631687.386636734 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631696.784683704 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631698.876183748 (ncf_main.py:288) eval_start : 3
-:::MLPv0.5.0 ncf 1541631707.906416178 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 3}
-:::MLPv0.5.0 ncf 1541631707.907561779 (ncf_main.py:298) eval_accuracy : {"value": 0.6261904835700989, "epoch": 3}
-:::MLPv0.5.0 ncf 1541631707.908753633 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 3}
-:::MLPv0.5.0 ncf 1541631707.909807205 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631707.910764217 (ncf_main.py:307) eval_stop : 3
-:::MLPv0.5.0 ncf 1541631707.912425280 (ncf_main.py:250) train_epoch : 4
-:::MLPv0.5.0 ncf 1541631710.637432575 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631710.859166145 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631710.860014200 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631710.861742735 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631711.817578554 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631711.818401098 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631711.819155693 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631720.856826782 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631723.175854445 (ncf_main.py:288) eval_start : 4
-:::MLPv0.5.0 ncf 1541631732.161775351 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 4}
-:::MLPv0.5.0 ncf 1541631732.162794590 (ncf_main.py:298) eval_accuracy : {"value": 0.6246091723442078, "epoch": 4}
-:::MLPv0.5.0 ncf 1541631732.163615465 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 4}
-:::MLPv0.5.0 ncf 1541631732.164470911 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631732.165247202 (ncf_main.py:307) eval_stop : 4
-:::MLPv0.5.0 ncf 1541631732.166882753 (ncf_main.py:250) train_epoch : 5
-:::MLPv0.5.0 ncf 1541631734.412759304 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631734.635779142 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631734.636681557 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631734.638389111 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631735.594851017 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631735.595717669 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631735.596507072 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631744.615598440 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631747.492086411 (ncf_main.py:288) eval_start : 5
-:::MLPv0.5.0 ncf 1541631756.363870144 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 5}
-:::MLPv0.5.0 ncf 1541631756.364969015 (ncf_main.py:298) eval_accuracy : {"value": 0.6296130418777466, "epoch": 5}
-:::MLPv0.5.0 ncf 1541631756.366026402 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 5}
-:::MLPv0.5.0 ncf 1541631756.367037296 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631756.367975235 (ncf_main.py:307) eval_stop : 5
-:::MLPv0.5.0 ncf 1541631756.369771481 (ncf_main.py:250) train_epoch : 6
-:::MLPv0.5.0 ncf 1541631758.621215105 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631758.845230579 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631758.846034765 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631758.847634554 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631759.813386679 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631759.814144373 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631759.814892054 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631768.872290850 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631771.607383251 (ncf_main.py:288) eval_start : 6
-:::MLPv0.5.0 ncf 1541631780.619389296 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 6}
-:::MLPv0.5.0 ncf 1541631780.620537758 (ncf_main.py:298) eval_accuracy : {"value": 0.6349273920059204, "epoch": 6}
-:::MLPv0.5.0 ncf 1541631780.621808290 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 6}
-:::MLPv0.5.0 ncf 1541631780.623294115 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631780.624791861 (ncf_main.py:307) eval_stop : 6
-:::MLPv0.5.0 ncf 1541631780.628558397 (ncf_main.py:250) train_epoch : 7
-:::MLPv0.5.0 ncf 1541631782.781100273 (data_async_generation.py:296) input_order 
-:::MLPv0.5.0 ncf 1541631783.003668547 (data_async_generation.py:314) input_size : 99385344
-:::MLPv0.5.0 ncf 1541631783.004613638 (data_async_generation.py:316) input_batch_size : 98304
-:::MLPv0.5.0 ncf 1541631783.006322145 (data_async_generation.py:347) input_order 
-:::MLPv0.5.0 ncf 1541631783.971688986 (data_async_generation.py:227) input_step_train_neg_gen 
-:::MLPv0.5.0 ncf 1541631783.972500324 (data_async_generation.py:229) input_hp_num_neg : 4
-:::MLPv0.5.0 ncf 1541631783.973114014 (data_async_generation.py:233) input_hp_sample_train_replacement : true
-:::MLPv0.5.0 ncf 1541631793.262183428 (data_async_generation.py:277) input_order 
-:::MLPv0.5.0 ncf 1541631795.851679325 (ncf_main.py:288) eval_start : 7
-:::MLPv0.5.0 ncf 1541631804.843798399 (ncf_main.py:296) eval_target : {"value": 0.635, "epoch": 7}
-:::MLPv0.5.0 ncf 1541631804.845178843 (ncf_main.py:298) eval_accuracy : {"value": 0.6357071995735168, "epoch": 7}
-:::MLPv0.5.0 ncf 1541631804.846249342 (ncf_main.py:301) eval_hp_num_neg : {"value": 999, "epoch": 7}
-:::MLPv0.5.0 ncf 1541631804.847424507 (ncf_main.py:305) eval_hp_num_users : 138493
-:::MLPv0.5.0 ncf 1541631804.848529577 (ncf_main.py:307) eval_stop : 7
-:::MLPv0.5.0 ncf 1541631804.850411892 (ncf_main.py:322) run_stop : {"success": true}
-:::MLPv0.5.0 ncf 1541631811.008860111 (ncf_main.py:328) run_final 
diff --git a/data_generation/fractal_graph_expansions/graph_analysis_test.py b/data_generation/fractal_graph_expansions/graph_analysis_test.py
deleted file mode 100644
index 943b36205..000000000
--- a/data_generation/fractal_graph_expansions/graph_analysis_test.py
+++ /dev/null
@@ -1,64 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for graph_analysis."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-
-import numpy as np
-import tensorflow as tf
-
-import graph_analysis
-from test_util import all_close
-from test_util import random_binary_sparse_matrix
-
-
-class GraphAnalysisTest(tf.test.TestCase):
-
-  def test_svd_is_valid(self):
-    np.random.seed(0)
-
-    num_rows = 16
-    num_cols = 32
-    num_non_zeros = 100
-    input_matrix = random_binary_sparse_matrix(
-        num_non_zeros, num_rows, num_cols)
-    k = 5
-
-    (u, s, v) = graph_analysis.sparse_svd(input_matrix, k, max_iter=16)
-
-    # Check that singular values are in increasing order:
-    for i in range(k - 1):
-      self.assertGreaterEqual(s[i + 1], s[i])
-
-    # Check that singular vector matrices have the right shapes.
-    self.assertEqual(u.shape[0], num_rows)
-    self.assertEqual(u.shape[1], k)
-
-    self.assertEqual(v.shape[0], k)
-    self.assertEqual(v.shape[1], num_cols)
-
-    # Check that u is column orthogonal.
-    self.assertTrue(all_close(np.matmul(u.T, u), np.eye(k)))
-
-    # Check that v is row orthogonal.
-    self.assertTrue(all_close(np.matmul(v, v.T), np.eye(k)))
-
-
-if __name__ == "__main__":
-  tf.test.main()
diff --git a/data_generation/fractal_graph_expansions/graph_expansion_test.py b/data_generation/fractal_graph_expansions/graph_expansion_test.py
deleted file mode 100644
index 548dd4034..000000000
--- a/data_generation/fractal_graph_expansions/graph_expansion_test.py
+++ /dev/null
@@ -1,146 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for graph_expansion."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import itertools
-
-
-
-import numpy as np
-import tensorflow as tf
-
-import graph_expansion
-from test_util import random_binary_sparse_matrix
-from test_util import read_from_serialized_file
-
-
-class GraphExpansionTest(tf.test.TestCase):
-
-  def test_produces_synthetic_interactions_with_right_shape(self):
-    np.random.seed(0)
-
-    left_matrix_num_rows = 4
-    left_matrix_num_cols = 8
-    left_matrix = np.ones((left_matrix_num_rows, left_matrix_num_cols))
-
-    right_matrix_num_rows = 16
-    right_matrix_num_cols = 32
-    right_matrix = random_binary_sparse_matrix(
-        50, right_matrix_num_rows,
-        right_matrix_num_cols) - random_binary_sparse_matrix(
-            50, right_matrix_num_rows, right_matrix_num_cols)
-    right_matrix = right_matrix.tocoo()
-    right_matrix_num_non_zeros = right_matrix.nnz
-
-    right_matrix_num_train = (right_matrix == 1).nnz
-    right_matrix_num_test = (right_matrix == -1).nnz
-
-    train_output_file = self.create_tempfile("temp_train.pkl")
-    test_output_file = self.create_tempfile("temp_test.pkl")
-
-    train_meta_output_file = self.create_tempfile("temp_train_meta.pkl")
-    test_meta_output_file = self.create_tempfile("temp_test_meta.pkl")
-
-    (metadata, train_metadata,
-     test_metadata) = graph_expansion.output_randomized_kronecker_to_pickle(
-         left_matrix, right_matrix,
-         train_output_file.full_path, test_output_file.full_path,
-         train_meta_output_file.full_path, test_meta_output_file.full_path,
-         remove_empty_rows=False)
-
-    # Left matrix is filled with 1s here.
-    self.assertEqual(metadata.num_interactions,
-                     left_matrix_num_rows * left_matrix_num_cols *
-                     right_matrix_num_non_zeros)
-    self.assertEqual(metadata.num_rows,
-                     left_matrix_num_rows * right_matrix_num_rows)
-    self.assertEqual(metadata.num_cols,
-                     left_matrix_num_cols * right_matrix_num_cols)
-
-    # Right matrix is filled with 1s here so there should be no test set.
-    self.assertEqual(train_metadata.num_interactions,
-                     left_matrix_num_rows * left_matrix_num_cols *
-                     right_matrix_num_train)
-    self.assertEqual(train_metadata.num_rows,
-                     left_matrix_num_rows * right_matrix_num_rows)
-    self.assertEqual(train_metadata.num_cols,
-                     left_matrix_num_cols * right_matrix_num_cols)
-
-    self.assertEqual(test_metadata.num_interactions,
-                     left_matrix_num_rows * left_matrix_num_cols *
-                     right_matrix_num_test)
-    self.assertEqual(test_metadata.num_rows,
-                     left_matrix_num_rows * right_matrix_num_rows)
-    self.assertEqual(test_metadata.num_cols,
-                     left_matrix_num_cols * right_matrix_num_cols)
-
-    pickled_train_metadata = read_from_serialized_file(
-        train_meta_output_file.full_path)
-    pickled_test_metadata = read_from_serialized_file(
-        test_meta_output_file.full_path)
-
-    self.assertEqual(train_metadata, pickled_train_metadata)
-    self.assertEqual(test_metadata, pickled_test_metadata)
-
-  def test_produces_synthetic_interactions_with_right_content(self):
-    np.random.seed(0)
-
-    left_matrix_num_rows = 4
-    left_matrix_num_cols = 8
-    left_matrix = np.ones((left_matrix_num_rows, left_matrix_num_cols))
-
-    right_matrix_num_rows = 16
-    right_matrix_num_cols = 32
-    right_matrix_num_non_zeros = 100
-    right_matrix = random_binary_sparse_matrix(
-        right_matrix_num_non_zeros, right_matrix_num_rows,
-        right_matrix_num_cols)
-    right_matrix = right_matrix.tocoo()
-
-    num_shards = len(left_matrix)
-    train_output_file = self.create_tempfile("temp_train.pkl")
-    train_output_shards = [
-        self.create_tempfile("temp_train.pkl_%d" % shard_idx)
-        for shard_idx in range(num_shards)]
-    test_output_file = self.create_tempfile("temp_test.pkl")
-
-    train_meta_output_file = self.create_tempfile("temp_train_meta.pkl")
-    test_meta_output_file = self.create_tempfile("temp_test_meta.pkl")
-
-    graph_expansion.output_randomized_kronecker_to_pickle(
-        left_matrix, right_matrix,
-        train_output_file.full_path, test_output_file.full_path,
-        train_meta_output_file.full_path, test_meta_output_file.full_path,
-        remove_empty_rows=False)
-
-    serialized_rows = []
-    for shard_output_file in train_output_shards:
-      serialized_rows.extend(read_from_serialized_file(
-          shard_output_file.full_path))
-
-    self.assertLen(serialized_rows,
-                   left_matrix_num_rows * right_matrix_num_rows)
-
-    output_item_set = set(itertools.chain(*serialized_rows))
-    self.assertEqual(output_item_set,
-                     set(range(left_matrix_num_cols * right_matrix_num_cols)))
-
-
-if __name__ == "__main__":
-  tf.test.main()
diff --git a/data_generation/fractal_graph_expansions/graph_reduction_test.py b/data_generation/fractal_graph_expansions/graph_reduction_test.py
deleted file mode 100644
index f8a70e805..000000000
--- a/data_generation/fractal_graph_expansions/graph_reduction_test.py
+++ /dev/null
@@ -1,101 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for graph_reduction."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-
-import numpy as np
-import tensorflow as tf
-
-import graph_reduction
-from test_util import all_close
-
-
-class GraphReductionTest(tf.test.TestCase):
-
-  def test_closest_column_orthogonal_matrix_is_column_orthogonal(self):
-    np.random.seed(0)
-
-    num_rows = 16
-    num_cols = 4
-
-    matrix = np.random.uniform(size=(num_rows, num_cols))
-
-    col_orth_matrix = graph_reduction._closest_column_orthogonal_matrix(matrix)
-
-    self.assertTrue(all_close(
-        np.matmul(col_orth_matrix.T, col_orth_matrix), np.eye(num_cols)))
-
-  def test_reduced_matrix_has_right_size(self):
-    np.random.seed(0)
-
-    num_rows = 64
-    num_cols = 16
-
-    reduced_num_rows = 12
-    reduced_num_cols = 3
-
-    svd_size = 8
-
-    u = np.random.uniform(size=(num_rows, svd_size))
-    v = np.random.uniform(size=(svd_size, num_cols))
-    s = np.random.uniform(size=svd_size)
-
-    resized_matrix = graph_reduction.resize_matrix(
-        (u, s, v), reduced_num_rows, reduced_num_cols)
-
-    self.assertEqual(resized_matrix.shape[0], reduced_num_rows)
-    self.assertEqual(resized_matrix.shape[1], reduced_num_cols)
-
-  def test_reduced_matrix_has_same_singular_value_spectrum(self):
-    np.random.seed(0)
-
-    num_rows = 64
-    num_cols = 16
-
-    reduced_num_rows = 12
-    reduced_num_cols = 3
-
-    svd_size = 8
-
-    u = np.random.uniform(size=(num_rows, svd_size))
-    v = np.random.uniform(size=(svd_size, num_cols))
-    s = sorted(np.random.uniform(size=svd_size))
-
-    resized_matrix = graph_reduction.resize_matrix(
-        (u, s, v), reduced_num_rows, reduced_num_cols)
-
-    resized_matrix_s = np.linalg.svd(resized_matrix, compute_uv=False)
-
-    self.assertTrue(all_close(s[::-1][:reduced_num_cols], resized_matrix_s))
-
-  def test_normalize_matrix(self):
-    np.random.seed(0)
-
-    matrix = np.random.normal(size=(16, 32))
-    normalized_matrix = graph_reduction.normalize_matrix(matrix)
-
-    self.assertEqual(matrix.shape, normalized_matrix.shape)
-
-    self.assertGreaterEqual(normalized_matrix.min(), 0.0)
-    self.assertLessEqual(normalized_matrix.max(), 1.0)
-
-
-if __name__ == "__main__":
-  tf.test.main()
diff --git a/data_generation/fractal_graph_expansions/random_matrix_ops_test.py b/data_generation/fractal_graph_expansions/random_matrix_ops_test.py
deleted file mode 100644
index 1111ec1cb..000000000
--- a/data_generation/fractal_graph_expansions/random_matrix_ops_test.py
+++ /dev/null
@@ -1,95 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for random_matrix_ops."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import collections
-
-
-
-import numpy as np
-import tensorflow as tf
-
-import random_matrix_ops
-from test_util import random_binary_sparse_matrix
-
-
-def _unique_value_counts(array):
-  return sorted(collections.Counter(array).values())
-
-
-class RandomMatrixOpsTest(tf.test.TestCase):
-
-  def test_dropout_coo(self):
-    np.random.seed(0)
-
-    dropout_rate = 0.5
-
-    num_rows = 16
-    num_cols = 32
-    num_non_zeros = 20
-
-    sparse_matrix = random_binary_sparse_matrix(num_non_zeros, num_rows,
-                                                num_cols)
-
-    sparse_coo_matrix = sparse_matrix.tocoo()
-
-    expected_num_sampled = int(num_non_zeros * dropout_rate)
-    sampled_coo_matrix = random_matrix_ops._dropout_sparse_coo_matrix(
-        sparse_coo_matrix,
-        dropout_rate,
-        min_dropout_rate=0.0,
-        max_dropout_rate=1.0)
-
-    self.assertEqual(sampled_coo_matrix.nnz, expected_num_sampled)
-
-  def test_shuffle_sparse_coo_matrix_preserves_unique_counts(self):
-    np.random.seed(0)
-
-    num_rows = 16
-    num_cols = 8
-    num_non_zeros = 20
-
-    sparse_matrix = random_binary_sparse_matrix(num_non_zeros, num_rows,
-                                                num_cols)
-
-    sparse_coo_matrix = sparse_matrix.tocoo()
-
-    # No values should be dropped out.
-    shuffled_sparse_coo_matrix = random_matrix_ops.shuffle_sparse_coo_matrix(
-        sparse_coo_matrix,
-        dropout_rate=0.0,
-        min_dropout_rate=0.0,
-        max_dropout_rate=1.0).tocoo()
-
-    original_sorted_row_counts = _unique_value_counts(sparse_coo_matrix.row)
-    original_sorted_col_counts = _unique_value_counts(sparse_coo_matrix.col)
-
-    shuffled_sorted_row_counts = _unique_value_counts(
-        shuffled_sparse_coo_matrix.row)
-    shuffled_sorted_col_counts = _unique_value_counts(
-        shuffled_sparse_coo_matrix.col)
-
-    self.assertEqual(
-        original_sorted_row_counts, shuffled_sorted_row_counts)
-    self.assertEqual(
-        original_sorted_col_counts, shuffled_sorted_col_counts)
-
-
-if __name__ == "__main__":
-  tf.test.main()
diff --git a/data_generation/fractal_graph_expansions/test_util.py b/data_generation/fractal_graph_expansions/test_util.py
deleted file mode 100644
index 91b77f469..000000000
--- a/data_generation/fractal_graph_expansions/test_util.py
+++ /dev/null
@@ -1,70 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Useful functions for the tests in this directory.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import cPickle as pickle
-import itertools
-
-
-
-import numpy as np
-import pandas as pd
-from scipy import sparse
-import tensorflow as tf
-
-
-def random_binary_sparse_matrix(num_non_zeros, num_rows, num_cols):
-  values = np.ones(num_non_zeros)
-  indices = list(itertools.product(range(num_rows), range(num_cols)))
-  sampled = np.random.choice(np.arange(num_rows * num_cols), num_non_zeros,
-                             replace=False)
-  sampled_indices = [indices[i] for i in sampled]
-  rows = [i for (i, _) in sampled_indices]
-  cols = [j for (_, j) in sampled_indices]
-  return sparse.csr_matrix((values, (rows, cols)), shape=[num_rows, num_cols])
-
-
-def sparse_to_df(sparse_matrix):
-  sparse_matrix_as_coo = sparse_matrix.tocoo()
-  row_indices = sparse_matrix_as_coo.row
-  col_indices = sparse_matrix_as_coo.col
-  data = sparse_matrix_as_coo.data
-
-  return pd.DataFrame({"row": row_indices, "col": col_indices, "data": data})
-
-
-def all_close(left_nd_array, right_nd_array, tol=1e-6):
-  return np.abs(left_nd_array - right_nd_array).max() <= tol
-
-
-def read_from_serialized_file(file_path):
-  with tf.gfile.Open(file_path, "rb") as infile:
-    return pickle.load(infile)
-
-
-def read_all_from_serialized_file(file_path):
-  objects = []
-  with tf.gfile.Open(file_path, "rb") as infile:
-    while True:
-      try:
-        objects.append(pickle.load(infile))
-      except EOFError:
-        break
-  return objects
diff --git a/data_generation/fractal_graph_expansions/test_util_test.py b/data_generation/fractal_graph_expansions/test_util_test.py
deleted file mode 100644
index 0dfc84530..000000000
--- a/data_generation/fractal_graph_expansions/test_util_test.py
+++ /dev/null
@@ -1,44 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for test_util."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-
-import tensorflow as tf
-
-import test_util
-
-
-class TestUtilTest(tf.test.TestCase):
-
-  def test_random_binary_sparse_matrix(self):
-    num_rows = 16
-    num_cols = 32
-    num_non_zeros = 100
-
-    sparse_matrix = test_util.random_binary_sparse_matrix(
-        num_non_zeros, num_rows, num_cols)
-
-    self.assertEqual(sparse_matrix.shape[0], num_rows)
-    self.assertEqual(sparse_matrix.shape[1], num_cols)
-    self.assertEqual(sparse_matrix.nnz, num_non_zeros)
-
-
-if __name__ == "__main__":
-  tf.test.main()
diff --git a/data_generation/fractal_graph_expansions/util_test.py b/data_generation/fractal_graph_expansions/util_test.py
deleted file mode 100644
index 937af1192..000000000
--- a/data_generation/fractal_graph_expansions/util_test.py
+++ /dev/null
@@ -1,57 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for util."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-
-
-import numpy as np
-import tensorflow as tf
-
-import util
-from test_util import random_binary_sparse_matrix
-from test_util import sparse_to_df
-
-
-class UtilTest(tf.test.TestCase):
-
-  def test_convert_df_to_sparse_matrix(self):
-    np.random.seed(0)
-
-    num_rows = 16
-    num_cols = 32
-    num_non_zeros = 40
-
-    sparse_matrix = random_binary_sparse_matrix(
-        num_non_zeros, num_rows, num_cols)
-
-    sparse_matrix_as_df = sparse_to_df(sparse_matrix)
-    sparse_matrix_as_df["data"] = 1.0
-
-    sparse_matrix_as_df_as_sparse = util.convert_df_to_sparse_matrix(
-        sparse_matrix_as_df, shape=(num_rows, num_cols))
-
-    self.assertEqual(sparse_matrix.shape, sparse_matrix_as_df_as_sparse.shape)
-
-    # Check that there are no non-zeros in the inequality boolean matrix.
-    self.assertEqual(
-        (sparse_matrix != sparse_matrix_as_df_as_sparse).nnz, 0)
-
-
-if __name__ == "__main__":
-  tf.test.main()
diff --git a/recommendation/pytorch/negative_sampling_cpp/test.py b/retired_benchmarks/ncf/negative_sampling_cpp/test.py
similarity index 100%
rename from recommendation/pytorch/negative_sampling_cpp/test.py
rename to retired_benchmarks/ncf/negative_sampling_cpp/test.py
diff --git a/image_classification/tensorflow/official/resnet/imagenet_test.py b/retired_benchmarks/resnet-tf1/official/resnet/imagenet_test.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/imagenet_test.py
rename to retired_benchmarks/resnet-tf1/official/resnet/imagenet_test.py
diff --git a/image_classification/tensorflow/official/resnet/layer_test.py b/retired_benchmarks/resnet-tf1/official/resnet/layer_test.py
similarity index 100%
rename from image_classification/tensorflow/official/resnet/layer_test.py
rename to retired_benchmarks/resnet-tf1/official/resnet/layer_test.py
diff --git a/image_classification/tensorflow/official/utils/arg_parsers/parsers_test.py b/retired_benchmarks/resnet-tf1/official/utils/arg_parsers/parsers_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/arg_parsers/parsers_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/arg_parsers/parsers_test.py
diff --git a/image_classification/tensorflow/official/utils/export/export_test.py b/retired_benchmarks/resnet-tf1/official/utils/export/export_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/export/export_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/export/export_test.py
diff --git a/image_classification/tensorflow/official/utils/logs/hooks_helper_test.py b/retired_benchmarks/resnet-tf1/official/utils/logs/hooks_helper_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/hooks_helper_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/hooks_helper_test.py
diff --git a/image_classification/tensorflow/official/utils/logs/hooks_test.py b/retired_benchmarks/resnet-tf1/official/utils/logs/hooks_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/hooks_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/hooks_test.py
diff --git a/image_classification/tensorflow/official/utils/logs/logger_test.py b/retired_benchmarks/resnet-tf1/official/utils/logs/logger_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/logger_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/logger_test.py
diff --git a/image_classification/tensorflow/official/utils/logs/metric_hook_test.py b/retired_benchmarks/resnet-tf1/official/utils/logs/metric_hook_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/logs/metric_hook_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/logs/metric_hook_test.py
diff --git a/image_classification/tensorflow/official/utils/misc/model_helpers_test.py b/retired_benchmarks/resnet-tf1/official/utils/misc/model_helpers_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/misc/model_helpers_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/misc/model_helpers_test.py
diff --git a/image_classification/tensorflow/official/utils/export/__init__.py b/retired_benchmarks/resnet-tf1/official/utils/testing/__init__.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/export/__init__.py
rename to retired_benchmarks/resnet-tf1/official/utils/testing/__init__.py
diff --git a/image_classification/tensorflow/official/utils/testing/integration.py b/retired_benchmarks/resnet-tf1/official/utils/testing/integration.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/integration.py
rename to retired_benchmarks/resnet-tf1/official/utils/testing/integration.py
diff --git a/image_classification/tensorflow/official/utils/testing/pylint.rcfile b/retired_benchmarks/resnet-tf1/official/utils/testing/pylint.rcfile
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/pylint.rcfile
rename to retired_benchmarks/resnet-tf1/official/utils/testing/pylint.rcfile
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data.py b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data.py
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data.py
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/dense/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/dense/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/reference_data_test/uniform_random/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/reference_data_test/uniform_random/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-1_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_projection_version-2_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-1_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_bottleneck_version-2_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-1_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_projection_version-2_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-1_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch-size-32_building_version-2_width-8_channels-4/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/expected_graph b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/expected_graph
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/expected_graph
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/expected_graph
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.data-00000-of-00001 b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.data-00000-of-00001
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.data-00000-of-00001
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.data-00000-of-00001
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.index b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.index
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.index
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/model.ckpt.index
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/results.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/results.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/results.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/results.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/tf_version.json b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/tf_version.json
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data/resnet/batch_norm/tf_version.json
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data/resnet/batch_norm/tf_version.json
diff --git a/image_classification/tensorflow/official/utils/testing/reference_data_test.py b/retired_benchmarks/resnet-tf1/official/utils/testing/reference_data_test.py
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/reference_data_test.py
rename to retired_benchmarks/resnet-tf1/official/utils/testing/reference_data_test.py
diff --git a/image_classification/tensorflow/official/utils/testing/scripts/presubmit.sh b/retired_benchmarks/resnet-tf1/official/utils/testing/scripts/presubmit.sh
similarity index 100%
rename from image_classification/tensorflow/official/utils/testing/scripts/presubmit.sh
rename to retired_benchmarks/resnet-tf1/official/utils/testing/scripts/presubmit.sh
diff --git a/translation/tensorflow/bert/modeling_test.py b/retired_benchmarks/transformer/tensorflow/bert/modeling_test.py
similarity index 100%
rename from translation/tensorflow/bert/modeling_test.py
rename to retired_benchmarks/transformer/tensorflow/bert/modeling_test.py
diff --git a/translation/tensorflow/bert/optimization_test.py b/retired_benchmarks/transformer/tensorflow/bert/optimization_test.py
similarity index 100%
rename from translation/tensorflow/bert/optimization_test.py
rename to retired_benchmarks/transformer/tensorflow/bert/optimization_test.py
diff --git a/translation/tensorflow/bert/tokenization_test.py b/retired_benchmarks/transformer/tensorflow/bert/tokenization_test.py
similarity index 100%
rename from translation/tensorflow/bert/tokenization_test.py
rename to retired_benchmarks/transformer/tensorflow/bert/tokenization_test.py
diff --git a/translation/tensorflow/transformer/compute_bleu_test.py b/retired_benchmarks/transformer/tensorflow/transformer/compute_bleu_test.py
similarity index 100%
rename from translation/tensorflow/transformer/compute_bleu_test.py
rename to retired_benchmarks/transformer/tensorflow/transformer/compute_bleu_test.py
diff --git a/translation/tensorflow/transformer/model/beam_search_test.py b/retired_benchmarks/transformer/tensorflow/transformer/model/beam_search_test.py
similarity index 100%
rename from translation/tensorflow/transformer/model/beam_search_test.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/beam_search_test.py
diff --git a/translation/tensorflow/transformer/model/model_utils_test.py b/retired_benchmarks/transformer/tensorflow/transformer/model/model_utils_test.py
similarity index 100%
rename from translation/tensorflow/transformer/model/model_utils_test.py
rename to retired_benchmarks/transformer/tensorflow/transformer/model/model_utils_test.py
diff --git a/translation/tensorflow/transformer/utils/tokenizer_test.py b/retired_benchmarks/transformer/tensorflow/transformer/utils/tokenizer_test.py
similarity index 100%
rename from translation/tensorflow/transformer/utils/tokenizer_test.py
rename to retired_benchmarks/transformer/tensorflow/transformer/utils/tokenizer_test.py