Repository: intel/e2eAIOK (language: python)

Issue:
Refine DLRM README



Discussion / hints:

Fix patch:
diff --git a/modelzoo/dlrm/README.md b/modelzoo/dlrm/README.md
index ff6d24935..881fce819 100644
--- a/modelzoo/dlrm/README.md
+++ b/modelzoo/dlrm/README.md
@@ -1,23 +1,62 @@
-# Notes
-This ls DLRM for ipex v0.2. You can use docker `xuechendi/oneapi-aikit:legacy_hydro.ai` to evaluate DLRM performance.
+# Intel® End-to-End AI Optimization Kit for DLRM
+## Original source disclose
+Source repo: https://github.com/facebookresearch/dlrm
 
-# Launch training with SDA
-`python run_e2eaiok.py --data_path "/home/vmagent/app/dataset/criteo" --model_name dlrm --conf conf/e2eaiok_defaults_dlrm_example.conf`
+---
 
-# For DLRM
-1. Use run_spark.sh to run spark ETL job, the data was saved on hdfs folder of /dlrm/input
-   Test and Validation data is under folder of /dlrm/output/test and validation
-   Then concat the generated small binary files under the folder of train, test, validation to train_data.bin, test_data.bin and val_data.bin 
-   And scp these three bin files under the same folder such as /mnt/DP_disk6/binary_dataset
-2. Install the dlrm conda env follow [Intel MLPerf DLRM Benchmark](https://github.com/mlperf/training_results_v0.7/tree/master/Intel/benchmarks/dlrm/1-node-4s-cpx-pytorch)
-3.Use dlrm/run_and_time.sh to run DLRM training
+# Quick Start
 
-------
-# For Model Compression on DLRM
-1. Use run_compress.sh to run model compress on DLRM, the configuration was saved as yaml in ./model_compress/
-2. Install the model compression running env:
-   
-   (1) Install the dlrm conda env as above;
+## Install
+```
+git clone https://github.com/intel/e2eAIOK.git
+git submodule update --init --recursive
+cd ${path_to_e2eaiok}/modelzoo/dlrm
+sh patch_dlrm.sh
+```
 
-   (2) Install the distiller follow the [link](https://teams.microsoft.com/l/file/52DEC602-9C2D-44CA-BC06-41D4850204B3?tenantId=46c98d88-e344-4ed4-8496-4ed7712e255d&fileType=docx&objectUrl=https%3A%2F%2Fintel.sharepoint.com%2Fsites%2FIAGS-SSP-SMPS-DPO-AnalyticsStorage%2FShared%20Documents%2FGeneral%2FNew%20Projects%2F2021_07%20Model%20Compression%2FDistiller%20Guide.docx&baseUrl=https%3A%2F%2Fintel.sharepoint.com%2Fsites%2FIAGS-SSP-SMPS-DPO-AnalyticsStorage&serviceName=teams&threadId=19:2fcb2b3c8b824e7ca5216b10d5624574@thread.skype&groupId=69adf55a-c293-4328-b4e3-bf0e344435e4);
+## Environment setup
+```
+cd ${path_to_e2eaiok}/Dockerfile-ubuntu18.04/
+docker build -t e2eaiok-pytorch . -f DockerfilePytorch
+```
 
+## Activate docker and conda
+```
+cd ${path_to_e2eaiok}
+docker run --shm-size=100g -it --privileged --network host --device=/dev/dri -v ${path_to_e2eaiok_dataset}:/home/vmagent/app/dataset -v `pwd`/:/home/vmagent/app/e2eaiok -w /home/vmagent/app/ e2eaiok-pytorch /bin/bash
+
+source /opt/intel/oneapi/intelpython/latest/envs/pytorch_mlperf/.local/env/setvars.sh
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/oneapi/intelpython/latest/envs/pytorch_mlperf/lib/python3.7/site-packages/torch_ipex-0.1-py3.7-linux-x86_64.egg/
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/oneapi/intelpython/latest/envs/pytorch_mlperf/lib/python3.7/site-packages/torch/lib/
+conda activate pytorch_mlperf
+
+cd /home/vmagent/app/e2eaiok/
+python setup.py install
+
+```
+
+## Download Dataset
+```
+Download the raw data files day_0.gz, ...,day_23.gz from https://labs.criteo.com/2013/12/download-terabyte-click-logs/ and unzip them
+
+```
+
+## RecDP Data Process
+```
+WIP to add
+```
+
+## Training
+```
+bash run_and_time_launch.sh
+```
+
+## Inference
+```
+bash inference.sh
+```
+
+## All In One
+```
+bash run_all_in_one.sh
+```
\ No newline at end of file
diff --git a/modelzoo/dlrm/dlrm.patch b/modelzoo/dlrm/dlrm.patch
index 2d4495882..67e0fdf09 100644
--- a/modelzoo/dlrm/dlrm.patch
+++ b/modelzoo/dlrm/dlrm.patch
@@ -1487,7 +1487,7 @@ index 0bda2ac..12d8d9f 100644
          else:
              sys.exit(
 diff --git a/dlrm_data_pytorch.py b/dlrm_data_pytorch.py
-index 852c577..15dac1d 100644
+index 852c577..2f188a2 100644
 --- a/dlrm_data_pytorch.py
 +++ b/dlrm_data_pytorch.py
 @@ -20,7 +20,6 @@ from __future__ import absolute_import, division, print_function, unicode_litera
@@ -1585,15 +1585,15 @@ index 852c577..15dac1d 100644
 -        ]
 -    )
 -
--
++def make_criteo_data_and_loaders(args):
+ 
 -def collate_wrapper_criteo_length(list_of_tuples):
 -    # where each tuple is (X_int, X_cat, y)
 -    transposed_data = list(zip(*list_of_tuples))
 -    X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
 -    X_cat = torch.tensor(transposed_data[1], dtype=torch.long)
 -    T = torch.tensor(transposed_data[2], dtype=torch.float32).view(-1, 1)
-+def make_criteo_data_and_loaders(args):
- 
+-
 -    batchSize = X_cat.shape[0]
 -    featureCnt = X_cat.shape[1]
 -
@@ -1695,7 +1695,84 @@ index 852c577..15dac1d 100644
          train_loader = torch.utils.data.DataLoader(
              train_data,
              batch_size=args.mini_batch_size,
-@@ -590,11 +542,6 @@ class RandomDataset(Dataset):
+@@ -572,6 +524,76 @@ def make_criteo_data_and_loaders(args, offset_to_length_converter=False):
+     return train_data, train_loader, test_data, test_loader
+ 
+ 
++def make_criteo_data_and_loaders_test(args):
++
++    if args.mlperf_logging and args.memory_map and args.data_set == "terabyte":
++        if args.mlperf_bin_loader:
++            test_file = args.eval_data_path
++            counts_file = args.day_feature_count
++            test_data = data_loader_terabyte.CriteoBinDataset(
++                data_file=test_file,
++                counts_file=counts_file,
++                batch_size=args.test_mini_batch_size,
++                max_ind_range=args.max_ind_range
++            )
++
++            mlperf_logger.log_event(key=mlperf_logger.constants.EVAL_SAMPLES,
++                                    value=test_data.num_samples)
++
++            test_loader = DataLoaderX(
++                test_data,
++                batch_size=None,
++                batch_sampler=None,
++                shuffle=False,
++                num_workers=0,
++                collate_fn=None,
++                pin_memory=False,
++                drop_last=False,
++            )
++        else:
++            data_filename = args.raw_data_file.split("/")[-1]
++            test_data = CriteoDataset(
++                args.data_set,
++                args.max_ind_range,
++                args.data_sub_sample_rate,
++                args.data_randomize,
++                "test",
++                args.raw_data_file,
++                args.processed_data_file,
++                args.memory_map
++            )
++            test_loader = data_loader_terabyte.DataLoader(
++                data_directory=data_directory,
++                data_filename=data_filename,
++                days=[23],
++                batch_size=args.test_mini_batch_size,
++                max_ind_range=args.max_ind_range,
++                split="test"
++            )
++    else:
++
++        test_data = CriteoDataset(
++            args.data_set,
++            args.max_ind_range,
++            args.data_sub_sample_rate,
++            args.data_randomize,
++            "test",
++            args.raw_data_file,
++            args.processed_data_file,
++            args.memory_map
++        )
++        test_loader = torch.utils.data.DataLoader(
++            test_data,
++            batch_size=args.test_mini_batch_size,
++            shuffle=False,
++            num_workers=args.test_num_workers,
++            collate_fn=collate_wrapper_criteo,
++            pin_memory=False,
++            drop_last=False,  # True
++        )
++
++    return test_data, test_loader
++
+ # uniform ditribution (input data)
+ class RandomDataset(Dataset):
+ 
+@@ -590,11 +612,6 @@ class RandomDataset(Dataset):
              trace_file="",
              enable_padding=False,
              reset_seed_on_access=False,
@@ -1707,7 +1784,7 @@ index 852c577..15dac1d 100644
              rand_seed=0
      ):
          # compute batch size
-@@ -619,11 +566,6 @@ class RandomDataset(Dataset):
+@@ -619,11 +636,6 @@ class RandomDataset(Dataset):
          self.enable_padding = enable_padding
          self.reset_seed_on_access = reset_seed_on_access
          self.rand_seed = rand_seed
@@ -1719,7 +1796,7 @@ index 852c577..15dac1d 100644
  
      def reset_numpy_seed(self, numpy_rand_seed):
          np.random.seed(numpy_rand_seed)
-@@ -648,17 +590,12 @@ class RandomDataset(Dataset):
+@@ -648,17 +660,12 @@ class RandomDataset(Dataset):
  
          # generate a batch of dense and sparse features
          if self.data_generation == "random":
@@ -1739,7 +1816,7 @@ index 852c577..15dac1d 100644
              )
          elif self.data_generation == "synthetic":
              (X, lS_o, lS_i) = generate_synthetic_input_batch(
-@@ -686,7 +623,7 @@ class RandomDataset(Dataset):
+@@ -686,7 +693,7 @@ class RandomDataset(Dataset):
          return self.num_batches
  
  
@@ -1748,7 +1825,7 @@ index 852c577..15dac1d 100644
      # where each tuple is (X, lS_o, lS_i, T)
      (X, lS_o, lS_i, T) = list_of_tuples[0]
      return (X,
-@@ -695,18 +632,7 @@ def collate_wrapper_random_offset(list_of_tuples):
+@@ -695,18 +702,7 @@ def collate_wrapper_random_offset(list_of_tuples):
              T)
  
  
@@ -1768,7 +1845,7 @@ index 852c577..15dac1d 100644
  
      train_data = RandomDataset(
          m_den,
-@@ -722,40 +648,8 @@ def make_random_data_and_loader(args, ln_emb, m_den,
+@@ -722,40 +718,8 @@ def make_random_data_and_loader(args, ln_emb, m_den,
          args.data_trace_file,
          args.data_trace_enable_padding,
          reset_seed_on_access=True,
@@ -1809,7 +1886,7 @@ index 852c577..15dac1d 100644
      train_loader = torch.utils.data.DataLoader(
          train_data,
          batch_size=1,
-@@ -765,17 +659,7 @@ def make_random_data_and_loader(args, ln_emb, m_den,
+@@ -765,17 +729,7 @@ def make_random_data_and_loader(args, ln_emb, m_den,
          pin_memory=False,
          drop_last=False,  # True
      )
@@ -1828,7 +1905,7 @@ index 852c577..15dac1d 100644
  
  
  def generate_random_data(
-@@ -791,7 +675,6 @@ def generate_random_data(
+@@ -791,7 +745,6 @@ def generate_random_data(
      data_generation="random",
      trace_file="",
      enable_padding=False,
@@ -1836,7 +1913,7 @@ index 852c577..15dac1d 100644
  ):
      nbatches = int(np.ceil((data_size * 1.0) / mini_batch_size))
      if num_batches != 0:
-@@ -815,8 +698,7 @@ def generate_random_data(
+@@ -815,8 +768,7 @@ def generate_random_data(
                  ln_emb,
                  n,
                  num_indices_per_lookup,
@@ -1846,7 +1923,7 @@ index 852c577..15dac1d 100644
              )
          elif data_generation == "synthetic":
              (Xt, lS_emb_offsets, lS_emb_indices) = generate_synthetic_input_batch(
-@@ -862,7 +744,6 @@ def generate_uniform_input_batch(
+@@ -862,7 +814,6 @@ def generate_uniform_input_batch(
      n,
      num_indices_per_lookup,
      num_indices_per_lookup_fixed,
@@ -1854,7 +1931,7 @@ index 852c577..15dac1d 100644
  ):
      # dense feature
      Xt = torch.tensor(ra.rand(n, m_den).astype(np.float32))
-@@ -890,71 +771,6 @@ def generate_uniform_input_batch(
+@@ -890,71 +841,6 @@ def generate_uniform_input_batch(
              r = ra.random(sparse_group_size)
              sparse_group = np.unique(np.round(r * (size - 1)).astype(np.int64))
              # reset sparse_group_size in case some index duplicates were removed
@@ -1926,7 +2003,7 @@ index 852c577..15dac1d 100644
              sparse_group_size = np.int64(sparse_group.size)
              # store lengths and indices
              lS_batch_offsets += [offset]
-@@ -1069,22 +885,21 @@ def trace_generate_lru(
+@@ -1069,22 +955,21 @@ def trace_generate_lru(
      max_sd = list_sd[-1]
      l = len(line_accesses)
      i = 0
@@ -1952,7 +2029,7 @@ index 852c577..15dac1d 100644
              line_accesses.append(line_ref)
          # save generated memory reference
          ztrace.append(mem_ref)
-@@ -1120,9 +935,9 @@ def trace_profile(trace, enable_padding=False):
+@@ -1120,9 +1005,9 @@ def trace_profile(trace, enable_padding=False):
      # number of elements in the array (assuming 1D)
      # n = trace.size
  
@@ -1965,7 +2042,7 @@ index 852c577..15dac1d 100644
      for x in trace:
          r = np.uint64(x / cache_line_size)
          l = len(rstack)
-@@ -1134,17 +949,17 @@ def trace_profile(trace, enable_padding=False):
+@@ -1134,17 +1019,17 @@ def trace_profile(trace, enable_padding=False):
              #          consecutive accesses (e.g. r, r) as 0 rather than 1.
              sd = l - i  # - 1
              # push r to the end of stack_distances
@@ -1989,7 +2066,7 @@ index 852c577..15dac1d 100644
  
      if enable_padding:
          # WARNING: notice that as the ratio between the number of samples (l)
-@@ -1176,7 +991,7 @@ def read_trace_from_file(file_path):
+@@ -1176,7 +1061,7 @@ def read_trace_from_file(file_path):
                  trace = list(map(lambda x: np.uint64(x), line.split(", ")))
              return trace
      except Exception:
@@ -1998,7 +2075,7 @@ index 852c577..15dac1d 100644
  
  
  def write_trace_to_file(file_path, trace):
-@@ -1186,7 +1001,7 @@ def write_trace_to_file(file_path, trace):
+@@ -1186,7 +1071,7 @@ def write_trace_to_file(file_path, trace):
                  np.array(trace).astype(np.uint64).tofile(f)
          else:
              with open(file_path, "w+") as f:
@@ -2007,7 +2084,7 @@ index 852c577..15dac1d 100644
                  f.write(s[1 : len(s) - 1])
      except Exception:
          print("ERROR: no output trace file has been provided")
-@@ -1197,7 +1012,7 @@ def read_dist_from_file(file_path):
+@@ -1197,7 +1082,7 @@ def read_dist_from_file(file_path):
          with open(file_path, "r") as f:
              lines = f.read().splitlines()
      except Exception:
@@ -2016,7 +2093,7 @@ index 852c577..15dac1d 100644
      # read unique accesses
      unique_accesses = [int(el) for el in lines[0].split(", ")]
      # read cumulative distribution (elements are passed as two separate lists)
-@@ -1211,19 +1026,20 @@ def write_dist_to_file(file_path, unique_accesses, list_sd, cumm_sd):
+@@ -1211,19 +1096,20 @@ def write_dist_to_file(file_path, unique_accesses, list_sd, cumm_sd):
      try:
          with open(file_path, "w") as f:
              # unique_acesses
@@ -2039,7 +2116,7 @@ index 852c577..15dac1d 100644
      import operator
      import argparse
  
-@@ -1269,7 +1085,7 @@ if __name__ == "__main__":
+@@ -1269,7 +1155,7 @@ if __name__ == "__main__":
      dist_sd = list(
          map(lambda tuple_x_k: tuple_x_k[1] / float(l), dc)
      )  # k = tuple_x_k[1]
@@ -2048,7 +2125,7 @@ index 852c577..15dac1d 100644
      for i, (_, k) in enumerate(dc):
          if i == 0:
              cumm_sd.append(k / float(l))
-@@ -1280,7 +1096,7 @@ if __name__ == "__main__":
+@@ -1280,7 +1166,7 @@ if __name__ == "__main__":
      ### write stack_distance and line_accesses to a file ###
      write_dist_to_file(args.dist_file, line_accesses, list_sd, cumm_sd)
  
@@ -2749,7 +2826,7 @@ index 8e3ed74..47b27d6 100644
          # Gather and ReducedSum ONNX operators, following the PyTorch approach.
          c2f = caffe2.python.onnx.frontend.Caffe2Frontend()
 diff --git a/dlrm_s_pytorch.py b/dlrm_s_pytorch.py
-index ec3394b..149ace0 100644
+index ec3394b..db5f5c7 100644
 --- a/dlrm_s_pytorch.py
 +++ b/dlrm_s_pytorch.py
 @@ -51,107 +51,64 @@
@@ -4223,7 +4300,7 @@ index ec3394b..149ace0 100644
              skip_upto_epoch = ld_k  # epochs
              skip_upto_batch = ld_j  # batches
          else:
-@@ -1418,460 +1177,395 @@ def run():
+@@ -1418,460 +1177,386 @@ def run():
              )
          )
          print(
@@ -4871,7 +4948,7 @@ index ec3394b..149ace0 100644
 +                    # .format(time_wrap(use_gpu) - accum_test_time_begin))
 +                    # if ext_dist.dist.get_rank()==0:
 +
-+                    file1 = open("/home/vmagent/app/hydro.ai/modelzoo/dlrm/dlrm/best_auc.txt",'w')
++                    file1 = open("./best_auc.txt",'w')
 +                    file1.writelines(str(best_auc_test))
 +                    file1.close()
 +
@@ -4990,11 +5067,8 @@ index ec3394b..149ace0 100644
 +                    test_end = time.time()
 +                    print(F"Test time:{test_end - test_start}")
 +                    ext_dist.barrier()
-+                    
-+                    # if (j>1) and (j+1)%13600 ==0:
-+                    #     print(F"Fineshed 13600 steps training")
-+                    #     break
-+
++                
++                
 +            mlperf_logger.barrier()
 +            mlperf_logger.log_end(key=mlperf_logger.constants.EPOCH_STOP,
 +                                  metadata={mlperf_logger.constants.EPOCH_NUM: k + 1})
@@ -5016,15 +5090,9 @@ index ec3394b..149ace0 100644
 +                "nbatches": nbatches,
 +                "nbatches_test": nbatches_test,
 +                "iter": j + 1,
-+                "state_dict": dlrm.state_dict(),
-+                "train_acc": gA,
-+                "train_loss": gL,
-+                "test_acc": gA_test,
-+                "test_loss": gL_test,
-+                "total_loss": total_loss,
-+                "total_accu": total_accu
++                "state_dict": dlrm.state_dict()
 +            },
-+            os.path.join(args.save_model, "dlrm_s_pytorch_" + str(dlrm.rank) + "_best.pkl"),
++            os.path.join(args.save_model, "dlrm_s_pytorch_" + str(dlrm.rank) + "_.pkl"),
          )
 -        # recover the model back
 -        dlrm_pytorch_onnx = onnx.load("dlrm_s_pytorch.onnx")
@@ -6726,10 +6794,10 @@ index 0000000..6e177ec
 +        onnx.checker.check_model(dlrm_pytorch_onnx)
 diff --git a/dlrm_s_pytorch_inference.py b/dlrm_s_pytorch_inference.py
 new file mode 100644
-index 0000000..6029b80
+index 0000000..575dabd
 --- /dev/null
 +++ b/dlrm_s_pytorch_inference.py
-@@ -0,0 +1,1125 @@
+@@ -0,0 +1,1107 @@
 +# Copyright (c) Facebook, Inc. and its affiliates.
 +#
 +# This source code is licensed under the MIT license found in the
@@ -7436,6 +7504,8 @@ index 0000000..6029b80
 +    # lamb
 +    parser.add_argument("--optimizer", type=int, default=0, help='optimizer:[0:sgd, 1:lamb/sgd, 2:adagrad, 3:sparseadam]')
 +    parser.add_argument("--lamblr", type=float, default=0.01, help='lr for lamb')
++    parser.add_argument("--eval-data-path", type=str, default="./data/valid.bin")
++    parser.add_argument("--day-feature-count", type=str, default="./data/day_fea_count.npz")
 +    args = parser.parse_args()
 +
 +    ext_dist.init_distributed(backend=args.dist_backend)
@@ -7712,24 +7782,8 @@ index 0000000..6029b80
 +
 +    # Load model is specified
 +    if not (args.load_model == ""):
-+        print("Loading saved model {}".format(args.load_model))
-+        if use_gpu:
-+            if dlrm.ndevices > 1:
-+                # NOTE: when targeting inference on multiple GPUs,
-+                # load the model as is on CPU or GPU, with the move
-+                # to multiple GPUs to be done in parallel_forward
-+                ld_model = torch.load(args.load_model)
-+            else:
-+                # NOTE: when targeting inference on single GPU,
-+                # note that the call to .to(device) has already happened
-+                ld_model = torch.load(
-+                    args.load_model,
-+                    map_location=torch.device('cuda')
-+                    # map_location=lambda storage, loc: storage.cuda(0)
-+                )
-+        else:
-+            # when targeting inference on CPU
-+            ld_model = torch.load(os.path.join(args.load_model, "dlrm_s_pytorch_" + str(dlrm.rank) + "_best.pkl"), map_location=torch.device('cpu'))
++        print("Loading trained model {}".format(args.load_model))
++        ld_model = torch.load(os.path.join(args.load_model, "dlrm_s_pytorch_" + str(dlrm.rank) + "_.pkl"), map_location=torch.device('cpu'))
 +        dlrm.load_state_dict(ld_model["state_dict"])
 +
 +    if args.use_ipex:
@@ -7849,11 +7903,7 @@ index 0000000..6029b80
 +            )
 +        )
 +    test_end = time.time()
-+    print(F"Test time:{test_end - test_start}")
-+    print(F"Total results length:{len(scores)}")
-+#     results = {'pred':scores.tolist(),'targets':targets.tolist()}
-+#     with open("results.json", "w") as f:
-+#         json.dump(results, f, sort_keys=True, indent=4)
++    print(F"Inference completed, time:{test_end - test_start}")
 +    ext_dist.barrier()
 diff --git a/extend_distributed.py b/extend_distributed.py
 index 1f2c8a5..8eaa16e 100644
@@ -10107,20 +10157,6 @@ index 83ab042..0000000
 -tqdm
 -torchrec-nightly
 -torchx-nightly
-diff --git a/run_and_time_launch.sh b/run_and_time_launch.sh
-new file mode 100755
-index 0000000..d6d2f95
---- /dev/null
-+++ b/run_and_time_launch.sh
-@@ -0,0 +1,8 @@
-+source ~/.local/env/setvars.sh
-+
-+seed_num=$(date +%s)
-+#export CCL_ATL_SHM=0
-+export KMP_BLOCKTIME=1
-+export KMP_AFFINITY="granularity=fine,compact,1,0"
-+
-+python  -u  ./launch.py --distributed --nproc_per_node=2 --nnodes=4 --hostfile ./hosts --master_addr="xxxxx" dlrm_s_pytorch.py  --max-ind-range=40000000  --data-generation=dataset --data-set=terabyte --raw-data-file=$DATA_PATH/day --processed-data-file=$DATA_PATH/terabyte_processed.npz --loss-function=bce --round-targets=True --num-workers=0 --test-num-workers=0 --use-ipex  --test-mini-batch-size=65536 --dist-backend=ccl  --mini-batch-size=262144 --print-freq=16 --print-time --optimizer=1 --bf16 --test-freq=800 --sparse-dense-boundary=403346  --memory-map --mlperf-logging --mlperf-auc-threshold=0.8025 --mlperf-bin-loader --mlperf-bin-shuffle --numpy-rand-seed=12345 $dlrm_extra_option 2>&1 | tee run_terabyte_${seed_num}.log
 diff --git a/run_compress.sh b/run_compress.sh
 new file mode 100755
 index 0000000..3766f37
diff --git a/modelzoo/dlrm/preproc/README.md b/modelzoo/dlrm/preproc/README.md
index 14932b79b..ac4e491c9 100644
--- a/modelzoo/dlrm/preproc/README.md
+++ b/modelzoo/dlrm/preproc/README.md
@@ -10,9 +10,9 @@
 * Make the directories as below or according to your settings for `convert_output_folder` in `spark_data_utils.py`. 
   This should be done for each spark worker node. Previous output data under these dirs should be dropped before re-processing.
 ```
-  /mnt/DP_disk5/test/
-  /mnt/DP_disk5/train/
-  /mnt/DP_disk5/validation/
+  ./test/
+  ./train/
+  ./validation/
 ```
 * Test raw data and validation raw data are generated by splitting `day_23`. They can be reused by copying to current HDFS or 
   manually uploaded to `test_input_folder/validation_input_folder` after the split.
diff --git a/modelzoo/dlrm/preproc/run_spark.sh b/modelzoo/dlrm/preproc/run_spark.sh
index 3df5f898f..ef59436f1 100644
--- a/modelzoo/dlrm/preproc/run_spark.sh
+++ b/modelzoo/dlrm/preproc/run_spark.sh
@@ -90,7 +90,7 @@ spark-submit --master $MASTER \
     --test_input_folder $temp_test \
     --validation_input_folder $temp_validation \
     --output_folder $OUTPUT_PATH/ \
-    --model_size_file /mnt/DP_disk7/model_size.json \
+    --model_size_file ./model_size.json \
    	--days 0-23 \
     --train_days 0-22 \
     --remain_days 23-23 \
diff --git a/modelzoo/dlrm/run_all_in_one.sh b/modelzoo/dlrm/run_all_in_one.sh
new file mode 100755
index 000000000..82198eab8
--- /dev/null
+++ b/modelzoo/dlrm/run_all_in_one.sh
@@ -0,0 +1,13 @@
+seed_num=$(date +%s)
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY="granularity=fine,compact,1,0"
+DATA_PATH=/home/vmagent/app/dataset/criteo
+MODEL_PATH=./result/
+mkdir $MODEL_PATH
+
+#train
+python  -u  ./dlrm/launch.py --distributed --nproc_per_node=2 ./dlrm/dlrm_s_pytorch.py --train-data-path="$DATA_PATH/train_data.bin" --eval-data-path="$DATA_PATH/test_data.bin" --day-feature-count="$DATA_PATH/day_fea_count.npz" --arch-sparse-feature-size=64 --arch-mlp-bot="13-128-64" --save-model=$MODEL_PATH --arch-mlp-top="256-128-1" --max-ind-range=40000000 --data-generation=dataset --data-set=terabyte --raw-data-file=$DATA_PATH/day --processed-data-file=$DATA_PATH/terabyte_processed.npz --loss-function=bce --round-targets=True --bf16 --num-workers=0 --test-num-workers=0 --use-ipex --optimizer=1 --dist-backend=ccl --learning-rate=16 --mini-batch-size=262144 --print-freq=16 --print-time --test-freq=16 --sparse-dense-boundary=403346 --test-mini-batch-size=131072 --lamblr=30 --lr-num-warmup-steps=4000 --lr-decay-start-step=5760 --lr-num-decay-steps=27000 --memory-map --mlperf-logging --mlperf-auc-threshold=0.9 --mlperf-bin-loader --mlperf-bin-shuffle --numpy-rand-seed=12345 $dlrm_extra_option 2>&1 | tee run_terabyte_train_${seed_num}.log
+#Inference
+python  -u  ./dlrm/launch.py --distributed --nproc_per_node=2 ./dlrm/dlrm_s_pytorch_inference.py --eval-data-path="$DATA_PATH/test_data.bin" --day-feature-count="$DATA_PATH/day_fea_count.npz" --load-model=$MODEL_PATH --arch-sparse-feature-size=64 --arch-mlp-bot="13-128-64" --arch-mlp-top="256-128-1" --max-ind-range=40000000 --data-generation=dataset --data-set=terabyte --raw-data-file=$DATA_PATH/day --processed-data-file=$DATA_PATH/terabyte_processed.npz --loss-function=bce --round-targets=True --bf16 --num-workers=0 --test-num-workers=0 --use-ipex --optimizer=1 --dist-backend=ccl --learning-rate=16 --mini-batch-size=262144 --print-freq=16 --print-time --test-freq=800 --sparse-dense-boundary=403346 --test-mini-batch-size=131072 --lamblr=30 --lr-num-warmup-steps=4000 --lr-decay-start-step=5760 --lr-num-decay-steps=27000 --memory-map --mlperf-logging --mlperf-auc-threshold=0.9 --mlperf-bin-loader --mlperf-bin-shuffle --numpy-rand-seed=12345 $dlrm_extra_option 2>&1 | tee run_terabyte_inference_${seed_num}.log
+#cat log
+cat run_terabyte_train_${seed_num}.log run_terabyte_inference_${seed_num}.log > run_terabyte_all_in_one_${seed_num}.log
\ No newline at end of file
diff --git a/modelzoo/dlrm/run_inference.sh b/modelzoo/dlrm/run_inference.sh
new file mode 100755
index 000000000..e561ea2d3
--- /dev/null
+++ b/modelzoo/dlrm/run_inference.sh
@@ -0,0 +1,8 @@
+seed_num=$(date +%s)
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY="granularity=fine,compact,1,0"
+seed_num=$(date +%s)
+DATA_PATH=/home/vmagent/app/dataset/criteo
+MODEL_PATH=./result/
+#Inference
+python  -u  ./dlrm/launch.py --distributed --nproc_per_node=2 ./dlrm/dlrm_s_pytorch_inference.py --eval-data-path="$DATA_PATH/test_data.bin" --day-feature-count="$DATA_PATH/day_fea_count.npz" --load-model=$MODEL_PATH --arch-sparse-feature-size=64 --arch-mlp-bot="13-128-64" --arch-mlp-top="256-128-1" --max-ind-range=40000000 --data-generation=dataset --data-set=terabyte --raw-data-file=$DATA_PATH/day --processed-data-file=$DATA_PATH/terabyte_processed.npz --loss-function=bce --round-targets=True --bf16 --num-workers=0 --test-num-workers=0 --use-ipex --optimizer=1 --dist-backend=ccl --learning-rate=16 --mini-batch-size=262144 --print-freq=16 --print-time --test-freq=800 --sparse-dense-boundary=403346 --test-mini-batch-size=131072 --lamblr=30 --lr-num-warmup-steps=4000 --lr-decay-start-step=5760 --lr-num-decay-steps=27000 --memory-map --mlperf-logging --mlperf-auc-threshold=0.9 --mlperf-bin-loader --mlperf-bin-shuffle --numpy-rand-seed=12345 $dlrm_extra_option 2>&1 | tee run_terabyte_inference_${seed_num}.log
\ No newline at end of file
diff --git a/modelzoo/dlrm/run_train.sh b/modelzoo/dlrm/run_train.sh
new file mode 100755
index 000000000..6bcc7dfac
--- /dev/null
+++ b/modelzoo/dlrm/run_train.sh
@@ -0,0 +1,10 @@
+seed_num=$(date +%s)
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY="granularity=fine,compact,1,0"
+
+DATA_PATH="/home/vmagent/app/dataset/criteo"
+MODEL_PATH="./result/"
+mkdir $MODEL_PATH
+
+#Inference
+python  -u  ./dlrm/launch.py --distributed --nproc_per_node=2 ./dlrm/dlrm_s_pytorch.py --train-data-path="$DATA_PATH/train_data.bin" --eval-data-path="$DATA_PATH/test_data.bin" --day-feature-count="$DATA_PATH/day_fea_count.npz" --arch-sparse-feature-size=64 --arch-mlp-bot="13-128-64" --save-model=$MODEL_PATH --arch-mlp-top="256-128-1" --max-ind-range=40000000 --data-generation=dataset --data-set=terabyte --raw-data-file=$DATA_PATH/day --processed-data-file=$DATA_PATH/terabyte_processed.npz --loss-function=bce --round-targets=True --bf16 --num-workers=0 --test-num-workers=0 --use-ipex --optimizer=1 --dist-backend=ccl --learning-rate=16 --mini-batch-size=262144 --print-freq=16 --print-time --test-freq=16 --sparse-dense-boundary=403346 --test-mini-batch-size=131072 --lamblr=30 --lr-num-warmup-steps=4000 --lr-decay-start-step=5760 --lr-num-decay-steps=27000 --memory-map --mlperf-logging --mlperf-auc-threshold=0.9 --mlperf-bin-loader --mlperf-bin-shuffle --numpy-rand-seed=12345 $dlrm_extra_option 2>&1 | tee run_terabyte_train_${seed_num}.log
\ No newline at end of file


Test patch:
diff --git a/e2eAIOK/SDA/modeladvisor/DLRMAdvisor.py b/e2eAIOK/SDA/modeladvisor/DLRMAdvisor.py
index 0d1cc55e9..a26c4436d 100644
--- a/e2eAIOK/SDA/modeladvisor/DLRMAdvisor.py
+++ b/e2eAIOK/SDA/modeladvisor/DLRMAdvisor.py
@@ -105,7 +105,7 @@ def train_model(self, args):
         else:
             self.dist_launch(args)
         self.training_time = time.time() - start_time
-        file1 = open("/home/vmagent/app/e2eaiok/modelzoo/dlrm/dlrm/best_auc.txt",'r')
+        file1 = open("./best_auc.txt",'r')
         lines = file1.readlines()
         file1.close()
         self.mean_accuracy = float(lines[-1])
diff --git a/tests/cicd/Jenkinsfile b/tests/cicd/Jenkinsfile
index 10e98cec4..70dffc16f 100644
--- a/tests/cicd/Jenkinsfile
+++ b/tests/cicd/Jenkinsfile
@@ -27,7 +27,7 @@ pipeline {
             steps {
                 sh 'docker run --rm --privileged --network host --device=/dev/dri -v /root/cicd_logs:/home/vmagent/app/cicd_logs -v /mnt/DP_disk1/dataset:/home/vmagent/app/dataset -v `pwd`:/home/vmagent/app/e2eaiok -w /home/vmagent/app/ e2eaiok-tensorflow /bin/bash -c "SIGOPT_API_TOKEN=$SIGOPT_API_TOKEN USE_SIGOPT=$USE_SIGOPT . /home/vmagent/app/e2eaiok/tests/cicd/jenkins_pipeline_test.sh"'
                 sh 'docker run --rm --privileged --network host --device=/dev/dri -v /root/cicd_logs:/home/vmagent/app/cicd_logs -v /mnt/DP_disk1/dataset:/home/vmagent/app/dataset -v `pwd`:/home/vmagent/app/e2eaiok -w /home/vmagent/app/ e2eaiok-tensorflow /bin/bash -c "SIGOPT_API_TOKEN=$SIGOPT_API_TOKEN USE_SIGOPT=$USE_SIGOPT . /home/vmagent/app/e2eaiok/tests/cicd/docker_horovod_test.sh"'
-                sh 'docker run --rm --privileged --network host --device=/dev/dri -v /root/cicd_logs:/home/vmagent/app/cicd_logs -v /mnt/DP_disk1/dataset:/home/vmagent/app/dataset -v `pwd`:/home/vmagent/app/e2eaiok -w /home/vmagent/app/ e2eaiok-pytorch110 /bin/bash -c "SIGOPT_API_TOKEN=$SIGOPT_API_TOKEN USE_SIGOPT=$USE_SIGOPT . /home/vmagent/app/e2eaiok/tests/cicd/docker_torchccl_test.sh"'
+                sh 'docker run --rm --privileged --network host --device=/dev/dri -v /root/cicd_logs:/home/vmagent/app/cicd_logs -v /mnt/DP_disk1/dataset:/home/vmagent/app/dataset -v `pwd`:/home/vmagent/app/e2eaiok -w /home/vmagent/app/ e2eaiok-pytorch /bin/bash -c "SIGOPT_API_TOKEN=$SIGOPT_API_TOKEN USE_SIGOPT=$USE_SIGOPT . /home/vmagent/app/e2eaiok/tests/cicd/docker_torchccl_test.sh"'
                 sh 'rm -f e2eaiok.db'
                 sh 'rm -rf result'
             }
diff --git a/tests/cicd/docker_torchccl_test.sh b/tests/cicd/docker_torchccl_test.sh
index 44266cd97..cba0de26d 100644
--- a/tests/cicd/docker_torchccl_test.sh
+++ b/tests/cicd/docker_torchccl_test.sh
@@ -1,8 +1,10 @@
 #!/bin/bash
 
-# enable oneAPI
-source /opt/intel/oneapi/setvars.sh --ccl-configuration=cpu_icc --force
+# init conda
+eval "$('/opt/intel/oneapi/intelpython/latest/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
+
 # UT check for docker env with torchccl
+source /opt/intel/oneapi/intelpython/latest/envs/pytorch_mlperf/.local/env/setvars.sh
 cd /home/vmagent/app/e2eaiok
-conda activate pytorch_1.10
+conda activate pytorch_mlperf
 mpirun -n 2 -l pytest -v --capture=tee-sys tests/cicd/src/test_docker_torchccl.py
\ No newline at end of file