Repository: intel/e2eAIOK (language: python)

Issue:
Update README.md



Discussion / hints:
1. revert code name e2eaiok. 
2. some other minor changes and typo fixes. 
@xuechendi  please review. 
LGTM

Fix patch:
diff --git a/example/instruction_tuning_pipeline/README.md b/example/instruction_tuning_pipeline/README.md
new file mode 100644
index 000000000..59b047b25
--- /dev/null
+++ b/example/instruction_tuning_pipeline/README.md
@@ -0,0 +1,152 @@
+# Deltatuner Fine-tuning Example
+
+This example demonstrates how to finetune the pretrained large language model (LLM) with the instruction-following dataset by using the Deltatuner and some foundamental models. Giving Fine-tuned model the textual instruction, it will respond with the textual response. This example have been validated on the 4th Gen Intel® Xeon® Processors, Sapphire Rapids.
+This example was based upon [Intel® Extension for Transformers](https://github.com/intel/intel-extension-for-transformers/tree/main/workflows/chatbot/fine_tuning).
+
+## Prerequisite​
+
+### 1. Environment​
+- build the docker image
+```shell
+docker build -f docker/Dockerfile -t chatbot_finetune .
+```
+- create docker container
+```shell
+# in cpu server
+docker run -it --name chatbot \
+    --privileged --network host --ipc=host \
+    --device=/dev/dri \
+    -v /dev/shm:/dev/shm \
+    -v ~:/home/vmagent/app \
+    -w /home/vmagent/app  \
+    chatbot_finetune:latest \
+    /bin/bash
+
+# in gpu server
+docker run -it --name chatbot \
+    --privileged --network host --ipc=host \
+    --device=/dev/dri \
+    --runtime=nvidia \
+    -v /dev/shm:/dev/shm \
+    -v /mnt/DP_disk1/dataset:/home/vmagent/app/data \
+    -v /mnt/DP_disk1/yu:/home/vmagent/app \
+    -w /home/vmagent/app/  \
+    chatbot_finetune:latest \
+    /bin/bash
+```
+- Or you can direct create from the bare mental env
+```shell
+pip install -r requirements.txt
+```
+
+- Install the `deltatuner` python package
+```shell
+pip install deltatuner
+```
+
+### 2. Prepare the Model
+
+- MPT: Download [the released model on Huggingface](https://huggingface.co/mosaicml/mpt-7b)
+- Llama-7B: Download [the released model on Huggingface](https://huggingface.co/huggyllama/llama-7b)
+- Llama-2-7B: Download [the released model on Huggingface](https://huggingface.co/meta-llama/Llama-2-7b-hf)
+
+### 3. Prepare Dataset
+The [Alpaca dataset](https://github.com/tatsu-lab/stanford_alpaca) from Stanford University as the general domain dataset to fine-tune the model. This dataset is provided in the form of a JSON file, [alpaca_data.json](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json). You can also download from [the released dataset on Huggingface](https://huggingface.co/datasets/tatsu-lab/alpaca), or you can download [the cleaned version of Alpaca on Huggingface](https://huggingface.co/datasets/yahma/alpaca-cleaned).
+
+
+## Finetune
+
+### 1. Single Node Fine-tuning in Xeon SPR
+
+#### Lora Fine-tuning with deltatuner
+
+For [MPT](https://huggingface.co/mosaicml/mpt-7b), use the below command line for finetuning on the Alpaca dataset. This model also requires that trust_remote_code=True be passed to the from_pretrained method. This is because we use a custom MPT model architecture that is not yet part of the Hugging Face transformers package.
+
+```bash
+# fine-tune mpt-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/mpt-7b" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/mpt_denas-lora_model \
+    --peft lora \
+    --delta lora \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/mpt-denas-lora-run-1epoch.log
+```
+
+Where the `--dataset_concatenation` argument is a way to vastly accelerate the fine-tuning process through training samples concatenation. With several tokenized sentences concatenated into a longer and concentrated sentence as the training sample instead of having several training samples with different lengths, this way is more efficient due to the parallelism characteristic provided by the more concentrated training samples.
+
+For finetuning on SPR, add `--bf16` argument will speedup the finetuning process without the loss of model's performance.
+
+For model profile, add `--profile` argument, the forward time and backward time will print on the console.
+
+For target modules of LoRA, use `--lora_target_modules` argument, the default target module of MPT and LLaMa model is `"llama": ["q_proj", "v_proj"],"mpt": ["Wqkv"]`. If you want to adapt to full modules on LLmMa model, please use `--lora_target_modules q_proj v_proj k_proj o_proj up_proj down_proj`, while it is `--lora_target_modules Wqkv out_proj up_proj down_proj` in MPT model.
+
+#### pure Lora Fine-tuning
+
+For fine-tune with Lora only algorighm, you can try the following command.
+```bash
+# fine-tune mpt-7b with lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/mpt-7b" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/mpt_lora_model \
+    --peft lora \
+    --denas False \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/mpt-lora-run-1epoch.log
+```
+
+## Evaluate the model
+
+For model evaluation, we follow the same method in [open_llm_leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), which evaluate 4 key benchmarks in the [Eleuther AI Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness).
+
+- To setup the evaluate env, use the following command
+```bash
+git clone https://github.com/EleutherAI/lm-evaluation-harness
+cd lm-evaluation-harness
+git checkout b281b0921b636bc3
+pip install -e .
+```
+
+- An example on evaluate on the hellaswag tasks
+```python
+python main.py --model hf-causal-experimental \
+    --model_args "pretrained=/home/vmagent/app/data/llama-7b,use_accelerate=True" \
+    --tasks hellaswag \
+    --num_fewshot 10 \
+    --batch_size auto --max_batch_size 16 \
+    --output_path /home/vmagent/app/data/llm-eval/test
+
+```
+Note: 
+- To view all the arguments that are supported by `--model_args`, please refer to [lm-evaluation-harness/HuggingFaceAutoLM](https://github.com/EleutherAI/lm-evaluation-harness/blob/b281b0921b636bc36ad05c0b0b0763bd6dd43463/lm_eval/models/huggingface.py#L60C7-L60C24).
+- For example evaluation on more benchmarks, please refer to the [`evaluate_benchmark` folder](../../tests/deltatuner/evaluate_benchmark/).
\ No newline at end of file
diff --git a/example/instruction_tuning_pipeline/finetune_clm.py b/example/instruction_tuning_pipeline/finetune_clm.py
new file mode 100755
index 000000000..eb27a3231
--- /dev/null
+++ b/example/instruction_tuning_pipeline/finetune_clm.py
@@ -0,0 +1,844 @@
+#!/usr/bin/env python
+# coding=utf-8
+
+# Apache v2 license
+# Copyright (C) 2022 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datasets
+import logging
+import os
+import sys
+import transformers
+from transformers.modeling_utils import unwrap_model
+from dataclasses import dataclass, field
+from datasets import load_dataset
+from peft import (
+    LoraConfig,
+    PromptEncoderConfig,
+    PrefixTuningConfig,
+    PromptTuningConfig,
+    TaskType,
+    get_peft_model,
+    get_peft_model_state_dict,
+    PeftModel
+)
+from peft.tuners.adaption_prompt import AdaptionPromptConfig
+from deltatuner import deltatuner, SSFConfig, DeltaTunerModel, DeltaTunerArguments
+from transformers import (
+    AutoConfig,
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    DataCollatorForSeq2Seq,
+    HfArgumentParser,
+    TrainingArguments,
+    Trainer,
+    set_seed,
+)
+from transformers.trainer_utils import is_main_process
+from typing import Optional, List, Union
+import copy
+import re
+import torch
+import importlib.util
+from transformers.utils.import_utils import is_optimum_available
+import wandb
+
+IGNORE_INDEX = -100
+
+os.environ["WANDB_DISABLED"] = "false"
+
+logger = logging.getLogger(__name__)
+
+
+def is_optimum_habana_available():
+    return is_optimum_available() and importlib.util.find_spec("optimum.habana") != None
+
+
+@dataclass
+class ModelArguments:
+    """
+    Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.
+    """
+
+    model_name_or_path: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "The model checkpoint for weights initialization."
+            "Don't set if you want to train a model from scratch."
+        },
+    )
+    config_name: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "Pretrained config name or path if not the same as model_name"
+        },
+    )
+    tokenizer_name: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "Pretrained tokenizer name or path if not the same as model_name"
+        },
+    )
+    cache_dir: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "Where do you want to store the pretrained models downloaded from huggingface.co"
+        },
+    )
+    use_fast_tokenizer: bool = field(
+        default=True,
+        metadata={
+            "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."
+        },
+    )
+    model_revision: str = field(
+        default="main",
+        metadata={
+            "help": "The specific model version to use (can be a branch name, tag name or commit id)."
+        },
+    )
+    use_auth_token: bool = field(
+        default=False,
+        metadata={
+            "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script "
+            "with private models)."
+        },
+    )
+    trust_remote_code: bool = field(
+        default=False,
+        metadata={
+            "help": "should enable when using custom model architecture that is not yet part of the Hugging Face transformers package like MPT)."
+        },
+    )
+
+
+@dataclass
+class DataArguments:
+    """
+    Arguments pertaining to what data we are going to input our model for training and eval.
+    """
+
+    dataset_name: Optional[str] = field(
+        default=None,
+        metadata={"help": "The name of the dataset to use (via the datasets library)."},
+    )
+    dataset_config_name: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "The configuration name of the dataset to use (via the datasets library)."
+        },
+    )
+    train_file: Optional[str] = field(
+        default=None, metadata={"help": "The input training data file (a text file)."}
+    )
+    validation_file: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."
+        },
+    )
+    max_seq_length: Optional[int] = field(
+        default=512,
+        metadata={
+            "help": "The maximum total input sequence length after tokenization. Sequences longer "
+            "than this will be truncated."
+        },
+    )
+    validation_split_percentage: Optional[int] = field(
+        default=0,
+        metadata={
+            "help": "The percentage of the train set used as validation set in case there's no validation split"
+        },
+    )
+    overwrite_cache: bool = field(
+        default=False,
+        metadata={"help": "Overwrite the cached preprocessed datasets or not."},
+    )
+    pad_to_max_length: bool = field(
+        default=False,
+        metadata={
+            "help": "Whether to pad all samples to `max_seq_length`. "
+            "If False, will pad the samples dynamically when batching to the maximum length in the batch."
+        },
+    )
+    max_train_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": "For debugging purposes or quicker training, truncate the number of training examples to this "
+            "value if set."
+        },
+    )
+    max_eval_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
+            "value if set."
+        },
+    )
+    keep_in_memory: bool = field(
+        default=False,
+        metadata={
+            "help": "Whether to keep in memory the loaded dataset. Defaults to False."
+        },
+    )
+    dataset_seed: int = field(
+        default=42,
+        metadata={
+            "help": "Seed to use in dataset processing, different seeds might yield different datasets. This seed and the seed in training arguments are not related"
+        },
+    )
+    dataset_cache_directory: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "Path to directory where the processed dataset will be saved. If path exists, try to load processed dataset from this path."
+        },
+    )
+    dataset_concatenation: Optional[bool] = field(
+        default=False,
+        metadata={
+            "help": "Whether to concatenate the sentence for more efficient training."
+        },
+    )
+
+
+@dataclass
+class FinetuneArguments:
+    """
+    Arguments of finetune we are going to apply on the model.
+    """
+
+    lora_rank: int = field(
+        default=8,
+        metadata={"help": "Rank parameter in the LoRA method."},
+    )
+    lora_alpha: int = field(
+        default=16,
+        metadata={"help": "Alpha parameter in the LoRA method."},
+    )
+    lora_dropout: float = field(
+        default=0.05,
+        metadata={"help": "Dropout parameter in the LoRA method."},
+    )
+    lora_target_modules: List[str] = field(
+        default_factory=lambda: None,
+        metadata={"help": "Target modules for the LoRA method."},
+    )
+    adapter_layers: int = field(
+        default=30,
+        metadata={"help": "adapter layer number in the LLaMA-adapter."},
+    )
+    adapter_len: int = field(
+        default=10,
+        metadata={
+            "help": "The length of the adaption prompt to insert in the LLaMA-adapter."
+        },
+    )
+    num_virtual_tokens: int = field(
+        default=10,
+        metadata={
+            "help": "The length of the vitrual tokens to insert in P-tuning/Prompt-tuning/Prefix-tuning"
+        },
+    )
+    ptun_hidden_size: int = field(
+        default=1024,
+        metadata={"help": "The encoder hidden size in P-tuning"},
+    )
+    peft: Optional[str] = field(
+        default="",
+        metadata={
+            "help": ("apply peft. default set to lora"),
+            "choices": ["llama_adapter", "lora", "ptun", "prefix", "prompt", ""],
+        },
+    )
+    resume_peft: Optional[str] = field(
+        default="",
+        metadata={"help": "the path of peft model to resume"},
+    )
+    delta: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": ("whether to use the deltatuner optimization"),
+            "choices": ["auto", "lora", "ssf"]
+        },
+    )
+    profile: bool = field(
+        default=False,
+        metadata={"help": "Profile the model to get the forward and backward time"}
+    )
+    train_on_inputs: bool = field(
+        default=True,
+        metadata={"help": "if False, masks out inputs in loss"},
+    )
+    habana: bool = field(
+        default=False,
+        metadata={"help": "if False, masks out inputs in loss"},
+    )
+    debugs: bool = field(
+        default=False,
+        metadata={"help": "for debug"},
+    )
+    save_merged_model: bool = field(
+        default=False,
+        metadata={"help": "save merged model"},
+    )
+
+PROMPT_DICT = {
+    "prompt_with_input": (
+        "Below is an instruction that describes a task, paired with an input that provides further context. "
+        "Write a response that appropriately completes the request.\n\n"
+        "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:"
+    ),
+    "prompt_without_input": (
+        "Below is an instruction that describes a task. "
+        "Write a response that appropriately completes the request.\n\n"
+        "### Instruction:\n{instruction}\n\n### Response:"
+    ),
+}
+
+
+def create_prompts(examples):
+    prompts = {}
+    prompts["source"] = []
+    prompts["target"] = []
+    for example in examples:
+        prompt_template = (
+            PROMPT_DICT["prompt_with_input"]
+            if example["input"] != ""
+            else PROMPT_DICT["prompt_without_input"]
+        )
+        source = prompt_template.format_map(example)
+        prompts["source"].append(source)
+        prompts["target"].append(example["output"])
+    return prompts
+
+
+def main():
+    # See all possible arguments in src/transformers/training_args.py
+    # or by passing the --help flag to this script.
+    # We now keep distinct sets of args, for a cleaner separation of concerns.
+    if not is_optimum_habana_available():
+        parser = HfArgumentParser(
+            (ModelArguments, DataArguments, TrainingArguments, FinetuneArguments, DeltaTunerArguments)
+        )
+    else:
+        from optimum.habana import GaudiTrainingArguments
+
+        parser = HfArgumentParser(
+            (ModelArguments, DataArguments, GaudiTrainingArguments, FinetuneArguments, DeltaTunerArguments)
+        )
+    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
+        # If we pass only one argument to the script and it's the path to a json file,
+        # let's parse it to get our arguments.
+        model_args, data_args, training_args, finetune_args, deltatuner_args = parser.parse_json_file(
+            json_file=os.path.abspath(sys.argv[1])
+        )
+    else:
+        (
+            model_args,
+            data_args,
+            training_args,
+            finetune_args,
+            deltatuner_args
+        ) = parser.parse_args_into_dataclasses()
+
+    print(f'finetune_args is \n {finetune_args}')
+
+    if finetune_args.habana:
+        if not is_optimum_habana_available():
+            raise ImportError(
+                "optimum habana is not installed. refer https://github.com/huggingface/optimum-habana"
+            )
+    # Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        handlers=[logging.StreamHandler(sys.stdout)],
+    )
+    logger.setLevel(
+        logging.INFO if is_main_process(training_args.local_rank) else logging.WARN
+    )
+
+    # Log on each process the small summary
+    b16 = training_args.fp16 or training_args.bf16
+    logger.warning(
+        f"Process rank: {training_args.local_rank}, device: {training_args.device}"
+        + f"\ndistributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {b16}"
+    )
+    # Set the verbosity to info of the Transformers logger (on main process only):
+    if is_main_process(training_args.local_rank):
+        transformers.utils.logging.set_verbosity_info()
+        transformers.utils.logging.enable_default_handler()
+        transformers.utils.logging.enable_explicit_format()
+    logger.info(f"Training/evaluation parameters {training_args}")
+
+    # Set seed before initializing model.
+    set_seed(training_args.seed)
+
+    # Load pretrained model and tokenizer
+    #
+    # Distributed training:
+    # The .from_pretrained methods guarantee that only one local process can concurrently
+    # download model & vocab.
+    config_kwargs = {
+        "cache_dir": model_args.cache_dir,
+        "revision": model_args.model_revision,
+        "use_auth_token": True if model_args.use_auth_token else None,
+        "trust_remote_code": True if model_args.trust_remote_code else None,
+    }
+    if model_args.config_name:
+        config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)
+    elif model_args.model_name_or_path:
+        config = AutoConfig.from_pretrained(
+            model_args.model_name_or_path, **config_kwargs
+        )
+    else:
+        raise ValueError("Please provide value for model_name_or_path or config_name.")
+
+    tokenizer_kwargs = {
+        "cache_dir": model_args.cache_dir,
+        "use_fast": model_args.use_fast_tokenizer,
+        "revision": model_args.model_revision,
+        "use_auth_token": True if model_args.use_auth_token else None,
+    }
+    if model_args.tokenizer_name:
+        tokenizer = AutoTokenizer.from_pretrained(
+            model_args.tokenizer_name, **tokenizer_kwargs
+        )
+    elif model_args.model_name_or_path:
+        tokenizer = AutoTokenizer.from_pretrained(
+            model_args.model_name_or_path, **tokenizer_kwargs
+        )
+    elif config.tokenizer_name:
+        tokenizer = AutoTokenizer.from_pretrained(
+            config.tokenizer_name, **tokenizer_kwargs
+        )
+    else:
+        raise ValueError(
+            "You are instantiating a new tokenizer from scratch. This is not supported by this script."
+            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
+        )
+
+    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
+    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
+    # (the dataset will be downloaded automatically from the datasets Hub).
+    #
+    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
+    # 'text' is found. You can easily tweak this behavior (see below).
+    #
+    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
+    # download the dataset.
+    if data_args.dataset_name is not None:
+        # Downloading and loading a dataset from the hub.
+        raw_datasets = load_dataset(
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            cache_dir=model_args.cache_dir,
+            use_auth_token=True if model_args.use_auth_token else None,
+            streaming=data_args.streaming,
+        )
+
+        if "validation" not in raw_datasets.keys() and training_args.do_eval:
+            raw_datasets["validation"] = load_dataset(
+                data_args.dataset_name,
+                data_args.dataset_config_name,
+                split=f"train[:{data_args.validation_split_percentage}%]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                streaming=data_args.streaming,
+            )
+            raw_datasets["train"] = load_dataset(
+                data_args.dataset_name,
+                data_args.dataset_config_name,
+                split=f"train[{data_args.validation_split_percentage}%:]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                streaming=data_args.streaming,
+            )
+    else:
+        data_files = {}
+        dataset_args = {}
+        if data_args.train_file is not None:
+            data_files["train"] = data_args.train_file
+        if data_args.validation_file is not None:
+            data_files["validation"] = data_args.validation_file
+        extension = (
+            data_args.train_file.split(".")[-1]
+            if data_args.train_file is not None
+            else data_args.validation_file.split(".")[-1]
+        )
+        if extension == "txt":
+            extension = "text"
+            dataset_args["keep_linebreaks"] = data_args.keep_linebreaks
+        raw_datasets = load_dataset(
+            extension,
+            data_files=data_files,
+            cache_dir=model_args.cache_dir,
+            use_auth_token=True if model_args.use_auth_token else None,
+            **dataset_args,
+        )
+
+        # If no validation data is there, validation_split_percentage will be used to divide the dataset.
+        if "validation" not in raw_datasets.keys() and training_args.do_eval:
+            raw_datasets["validation"] = load_dataset(
+                extension,
+                data_files=data_files,
+                split=f"train[:{data_args.validation_split_percentage}%]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                **dataset_args,
+            )
+            raw_datasets["train"] = load_dataset(
+                extension,
+                data_files=data_files,
+                split=f"train[{data_args.validation_split_percentage}%:]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                **dataset_args,
+            )
+
+    # Preprocessing the datasets.
+    for key in raw_datasets:
+        prompts = create_prompts(raw_datasets[key])
+        columns_to_be_removed = list(raw_datasets[key].features.keys())
+        raw_datasets[key] = raw_datasets[key].add_column(
+            "prompt_sources", prompts["source"]
+        )
+        raw_datasets[key] = raw_datasets[key].add_column(
+            "prompt_targets", prompts["target"]
+        )
+        raw_datasets[key] = raw_datasets[key].remove_columns(columns_to_be_removed)
+
+    # Load model
+    if model_args.model_name_or_path:
+        model_dtype = torch.bfloat16 if training_args.bf16 else None
+        model = AutoModelForCausalLM.from_pretrained(
+            model_args.model_name_or_path,
+            from_tf=bool(".ckpt" in model_args.model_name_or_path),
+            config=config,
+            cache_dir=model_args.cache_dir,
+            revision=model_args.model_revision,
+            use_auth_token=True if model_args.use_auth_token else None,
+            trust_remote_code=True if model_args.trust_remote_code else None,
+            torch_dtype=model_dtype,
+            low_cpu_mem_usage=True,
+        )
+    else:
+        raise ValueError(
+            "Must provide model_name_or_path to load a pretrained CausalLM model."
+        )
+
+    if re.search("llama", model.config.architectures[0], re.IGNORECASE):
+        # unwind broken decapoda-research config
+        model.generation_config.pad_token_id = 0
+        model.generation_config.bos_token_id = 1
+        model.generation_config.eos_token_id = 2
+
+    if (
+        hasattr(model.generation_config, "pad_token_id")
+        and model.generation_config.pad_token_id is not None
+    ):
+        tokenizer.pad_token_id = model.generation_config.pad_token_id
+    if (
+        hasattr(model.generation_config, "eos_token_id")
+        and model.generation_config.eos_token_id is not None
+    ):
+        tokenizer.eos_token_id = model.generation_config.eos_token_id
+    if (
+        hasattr(model.generation_config, "bos_token_id")
+        and model.generation_config.bos_token_id is not None
+    ):
+        tokenizer.bos_token_id = model.generation_config.bos_token_id
+
+    if tokenizer.pad_token_id is None:
+        tokenizer.pad_token_id = tokenizer.eos_token_id
+    tokenizer.padding_side = "left"  # Allow batched inference
+
+    def tokenize(prompt, add_eos_token=True):
+        results = tokenizer(
+            prompt,
+            truncation=True,
+            max_length=data_args.max_seq_length,
+            padding=False,
+            return_tensors=None,
+        )
+        for i in range(len(results["input_ids"])):
+            if (
+                results["input_ids"][i][-1] != tokenizer.eos_token_id
+                and len(results["input_ids"][i]) < data_args.max_seq_length
+                and add_eos_token
+            ):
+                results["input_ids"][i].append(tokenizer.eos_token_id)
+                results["attention_mask"][i].append(1)
+
+        results["labels"] = copy.deepcopy(results["input_ids"])
+        results["input_id_len"] = [len(result) for result in results["input_ids"]]
+        return results
+
+    def preprocess_function(examples):
+        st = [
+            s + t
+            for s, t in zip(examples["prompt_sources"], examples["prompt_targets"])
+        ]
+        examples_tokenized = tokenize(st)
+        input_ids = examples_tokenized["input_ids"]
+        labels = examples_tokenized["labels"]
+        if not finetune_args.train_on_inputs:
+            sources_tokenized = tokenize(
+                examples["prompt_sources"], add_eos_token=False
+            )
+            for label, source_len in zip(labels, sources_tokenized["input_id_len"]):
+                label[:source_len] = [IGNORE_INDEX] * source_len
+        return dict(
+            input_ids=input_ids,
+            labels=labels,
+            attention_mask=examples_tokenized["attention_mask"],
+        )
+
+    with training_args.main_process_first(desc="dataset map pre-processing"):
+        tokenized_datasets = raw_datasets.map(
+            preprocess_function,
+            batched=True,
+            load_from_cache_file=not data_args.overwrite_cache,
+        )
+
+    if data_args.dataset_concatenation:
+
+        def concatenate_data(dataset, max_seq_length):
+            concatenated_dataset = {}
+            for column in dataset.features:
+                concatenated_data = [
+                    item for sample in dataset[column] for item in sample
+                ]
+                reshaped_data = [
+                    concatenated_data[i * max_seq_length : (i + 1) * max_seq_length]
+                    for i in range(len(concatenated_data) // max_seq_length)
+                ]
+                concatenated_dataset[column] = reshaped_data
+            return datasets.Dataset.from_dict(concatenated_dataset)
+
+        tokenized_datasets_ = tokenized_datasets["train"].remove_columns(
+            ["prompt_sources", "prompt_targets"]
+        )
+        tokenized_datasets["train"] = concatenate_data(
+            tokenized_datasets_, data_args.max_seq_length
+        )
+
+    if training_args.do_train:
+        if "train" not in tokenized_datasets:
+            raise ValueError("--do_train requires a train dataset")
+        train_dataset = tokenized_datasets["train"]
+        if data_args.max_train_samples is not None:
+            train_dataset = train_dataset.select(range(data_args.max_train_samples))
+
+    if training_args.do_eval:
+        if "validation" not in tokenized_datasets:
+            raise ValueError("--do_eval requires a validation dataset")
+        eval_dataset = tokenized_datasets["validation"]
+        if data_args.max_eval_samples is not None:
+            eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
+
+    # Data collator
+    # This one will take care of randomly masking the tokens.
+    data_collator = DataCollatorForSeq2Seq(
+        tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True
+    )
+    logger.info(
+        "Using data collator of type {}".format(data_collator.__class__.__name__)
+    )
+
+    if finetune_args.peft:
+        # PEFT settings
+        if finetune_args.peft == "lora":
+            peft_config = LoraConfig(
+                r=finetune_args.lora_rank,
+                lora_alpha=finetune_args.lora_alpha,
+                lora_dropout=finetune_args.lora_dropout,
+                target_modules=finetune_args.lora_target_modules,
+                bias="none",
+                task_type=TaskType.CAUSAL_LM,
+            )
+        elif finetune_args.peft == "llama_adapter":
+            peft_config = AdaptionPromptConfig(
+                adapter_layers=finetune_args.adapter_layers,
+                adapter_len=finetune_args.adapter_len,
+                task_type="CAUSAL_LM",
+            )
+        elif finetune_args.peft == "ptun":
+            peft_config = PromptEncoderConfig(
+                num_virtual_tokens=finetune_args.num_virtual_tokens,
+                encoder_hidden_size=finetune_args.ptun_hidden_size,
+                task_type="CAUSAL_LM",
+            )
+        elif finetune_args.peft == "prefix":
+            peft_config = PrefixTuningConfig(
+                num_virtual_tokens=finetune_args.num_virtual_tokens,
+                task_type="CAUSAL_LM",
+            )
+        elif finetune_args.peft == "prompt":
+            peft_config = PromptTuningConfig(
+                num_virtual_tokens=finetune_args.num_virtual_tokens,
+                task_type="CAUSAL_LM",
+            )
+
+        if finetune_args.resume_peft != "":
+            model = PeftModel.from_pretrained(model, finetune_args.resume_peft, config=peft_config)
+        else:
+            model = get_peft_model(model, peft_config)
+        logger.info("***original optimized model parameter***")
+        model.print_trainable_parameters()
+
+    if finetune_args.delta:
+        if finetune_args.resume_peft != "":
+            if finetune_args.delta == "ssf":
+                peft_config = SSFConfig(target_modules=deltatuner_args.ssf_target_modules,
+                bias="none",
+                task_type="CAUSAL_LM",
+                )
+            model = DeltaTunerModel.from_pretrained(model, finetune_args.resume_peft, config=peft_config, denas_config=deltatuner_args)
+        else:
+            model = deltatuner.optimize(model, tokenizer, algo=finetune_args.delta, deltatuning_args=deltatuner_args)
+        logger.info("***deltatuner optimized model parameter***")
+        model.print_trainable_parameters()
+         
+    if finetune_args.debugs:
+        if training_args.do_train:
+            train_dataset = train_dataset.select(range(8))
+        if training_args.do_eval:
+            eval_dataset = eval_dataset.select(range(8))
+
+    if not finetune_args.habana:
+        # Initialize our Trainer
+        trainer = Trainer(
+            model=model,
+            args=training_args,
+            train_dataset=train_dataset if training_args.do_train else None,
+            eval_dataset=eval_dataset if training_args.do_eval else None,
+            tokenizer=tokenizer,
+            data_collator=data_collator,
+        )
+    else:
+        from optimum.habana import GaudiConfig, GaudiTrainer
+
+        gaudi_config = GaudiConfig()
+        gaudi_config.use_fused_adam = True
+        gaudi_config.use_fused_clip_norm = True
+        # Initialize our Trainer
+        trainer = GaudiTrainer(
+            model=model,
+            gaudi_config=gaudi_config,
+            args=training_args,
+            train_dataset=train_dataset if training_args.do_train else None,
+            eval_dataset=eval_dataset if training_args.do_eval else None,
+            tokenizer=tokenizer,
+            data_collator=data_collator,
+        )
+
+    if finetune_args.profile:
+        profile_model(model, train_dataset, data_collator, training_args)
+
+    if training_args.do_train:
+        logger.info("*** Training ***")
+        trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
+        with training_args.main_process_first(desc="save model"):
+            if is_main_process(training_args.local_rank):
+                unwrapped_model = unwrap_model(model)
+                unwrapped_model.save_pretrained(
+                    training_args.output_dir, state_dict=unwrapped_model.state_dict()
+                )
+
+    # Evaluation
+    if training_args.do_eval:
+        logger.info("*** Evaluate ***")
+        metrics = trainer.evaluate()
+
+        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+        metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
+        
+        total_token = sum(eval_dataset.map(lambda example: {"sentence_len": len(example["input_ids"])})["sentence_len"][:])
+        metrics["eval_tokens"] = total_token
+
+        trainer.log_metrics("eval", metrics)
+        trainer.save_metrics("eval", metrics)
+
+    if finetune_args.save_merged_model:
+        if isinstance(model, PeftModel):
+            merged_model = model.merge_and_unload()
+            os.makedirs(os.path.join(training_args.output_dir,"merged_model"),exist_ok=True)
+            torch.save(merged_model.state_dict(),os.path.join(training_args.output_dir,"merged_model","pytorch_model.bin"))
+            print(f"Save merged model to {training_args.output_dir}")
+
+def profile_model(model, train_dataset, data_collator, args):
+    from transformers import get_scheduler
+    from torch.utils.data import DataLoader
+
+    saving_dir_name = 'hf-training-torch/tmp'
+
+    model.train()
+    train_dataset = train_dataset.select(range(100))
+
+    train_dataloader = DataLoader(
+        train_dataset, shuffle=True, batch_size=args.per_device_train_batch_size, collate_fn=data_collator
+    )
+
+    # Optimizer
+    # Split weights in two groups, one with weight decay and the other not.
+    no_decay = ["bias", "LayerNorm.weight"]
+    optimizer_grouped_parameters = [
+        {
+            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
+            "weight_decay": args.weight_decay,
+        },
+        {
+            "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
+            "weight_decay": 0.0,
+        },
+    ]
+    optimizer = torch.optim.AdamW(optimizer_grouped_parameters, lr=args.learning_rate)
+
+    lr_scheduler = get_scheduler(
+        name=args.lr_scheduler_type,
+        optimizer=optimizer,
+        num_warmup_steps=0,
+        num_training_steps=args.num_train_epochs * len(train_dataloader),
+    )
+
+    # torch.profiler.ProfilerActivity.CUDA
+    # profile_memory=True,
+    # schedule=torch.profiler.schedule(skip_first=3, wait=1, warmup=1, active=2, repeat=2),
+    with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU], 
+                            on_trace_ready=torch.profiler.tensorboard_trace_handler(saving_dir_name),
+                            schedule=torch.profiler.schedule(skip_first=1, wait=1, warmup=3, active=7, repeat=1),
+    ) as prof:
+        with torch.profiler.record_function("model_all"):
+
+            for batch in train_dataloader:
+                batch = {k: v.to(args.device) for k, v in batch.items()}
+                with torch.profiler.record_function("model_inference"):
+                    outputs = model(**batch)
+                loss = outputs.loss
+                loss.backward()
+
+                optimizer.step()
+                lr_scheduler.step()
+                optimizer.zero_grad()
+                prof.step()
+    logger.info(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))
+    
+    sys.exit(0)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/example/instruction_tuning_pipeline/finetune_seq2seq.py b/example/instruction_tuning_pipeline/finetune_seq2seq.py
new file mode 100644
index 000000000..fd56b3bf8
--- /dev/null
+++ b/example/instruction_tuning_pipeline/finetune_seq2seq.py
@@ -0,0 +1,624 @@
+#!/usr/bin/env python
+# coding=utf-8
+
+# Apache v2 license
+# Copyright (C) 2022 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import math
+import os
+import sys
+from dataclasses import dataclass, field
+from itertools import chain
+from typing import Optional, List
+import numpy as np
+import datasets
+import evaluate
+import torch
+from datasets import load_dataset
+
+import transformers
+
+from transformers import (
+    HfArgumentParser,
+    Seq2SeqTrainer,
+    Seq2SeqTrainingArguments,
+    AutoTokenizer,
+    AutoConfig,
+    AutoModelForSeq2SeqLM,
+    set_seed
+)
+from peft import (
+    get_peft_model,
+    LoraConfig,
+    PrefixTuningConfig,
+    PromptEncoderConfig,
+    PromptTuningConfig,
+    TaskType,
+    prepare_model_for_int8_training,
+    PeftModel,
+    PeftConfig
+)
+from transformers import DataCollatorForSeq2Seq
+
+logger = logging.getLogger(__name__)
+
+PROMPT_DICT = {
+    "prompt_input": (
+        "Below is an instruction that describes a task, paired with an input that provides further context. "
+        "Write a response that appropriately completes the request.\n\n"
+        "### Instruction:\n{}\n\n### Input:\n{}\n\n### Response:"
+    ),
+    "prompt_no_input": (
+        "Below is an instruction that describes a task. "
+        "Write a response that appropriately completes the request.\n\n"
+        "### Instruction:\n{}\n\n### Response:"
+    ),
+}
+
+import nltk
+nltk.download("punkt", quiet=True)
+
+def postprocess_text(preds, labels):
+    preds = [pred.strip() for pred in preds]
+    labels = [label.strip() for label in labels]
+
+    # rougeLSum expects newline after each sentence
+    preds = ["\n".join(nltk.sent_tokenize(pred)) for pred in preds]
+    labels = ["\n".join(nltk.sent_tokenize(label)) for label in labels]
+
+    return preds, labels
+
+@dataclass
+class ModelArguments:
+    """
+    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
+    """
+
+    model_name_or_path: str = field(
+        metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"}
+    )
+    config_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
+    )
+    tokenizer_name: Optional[str] = field(
+        default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
+    )
+    cache_dir: Optional[str] = field(
+        default=None,
+        metadata={"help": "Where to store the pretrained models downloaded from huggingface.co"},
+    )
+    use_fast_tokenizer: bool = field(
+        default=True,
+        metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
+    )
+    model_revision: str = field(
+        default="main",
+        metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
+    )
+    use_auth_token: bool = field(
+        default=False,
+        metadata={
+            "help": (
+                "Will use the token generated when running `huggingface-cli login` (necessary to use this script "
+                "with private models)."
+            )
+        },
+    )
+
+@dataclass
+class DataTrainingArguments:
+    """
+    Arguments pertaining to what data we are going to input our model for training and eval.
+    """
+
+    dataset_name: Optional[str] = field(
+        default=None, metadata={"help": "The name of the dataset to use (via the datasets library)."}
+    )
+    dataset_config_name: Optional[str] = field(
+        default=None, metadata={"help": "The configuration name of the dataset to use (via the datasets library)."}
+    )
+    train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."})
+    validation_file: Optional[str] = field(
+        default=None,
+        metadata={"help": "An optional input evaluation data file to evaluate the perplexity on (a text file)."},
+    )
+    max_train_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": (
+                "For debugging purposes or quicker training, truncate the number of training examples to this "
+                "value if set."
+            )
+        },
+    )
+    max_eval_samples: Optional[int] = field(
+        default=None,
+        metadata={
+            "help": "For debugging purposes or quicker training, truncate the number of evaluation examples to this "
+            "value if set."
+        },
+    )
+
+    max_source_length: Optional[int] = field(
+        default=512,
+        metadata={
+            "help": (
+                "The maximum total input sequence length after tokenization. Sequences longer "
+                "than this will be truncated, sequences shorter will be padded."
+            )
+        },
+    )
+    max_target_length: Optional[int] = field(
+        default=256,
+        metadata={
+            "help": (
+                "The maximum total sequence length for target text after tokenization. Sequences longer "
+                "than this will be truncated, sequences shorter will be padded."
+            )
+        },
+    )
+
+    streaming: bool = field(default=False, metadata={"help": "Enable streaming mode"})
+    overwrite_cache: bool = field(
+        default=False, metadata={"help": "Overwrite the cached training and evaluation sets"}
+    )
+    validation_split_percentage: Optional[int] = field(
+        default=1,
+        metadata={
+            "help": "The percentage of the train set used as validation set in case there's no validation split"
+        },
+    )
+    preprocessing_num_workers: Optional[int] = field(
+        default=None,
+        metadata={"help": "The number of processes to use for the preprocessing."},
+    )
+    def __post_init__(self):
+        if self.streaming:
+            require_version("datasets>=2.0.0", "The streaming feature requires `datasets>=2.0.0`")
+
+        if self.dataset_name is None and self.train_file is None and self.validation_file is None:
+            raise ValueError("Need either a dataset name or a training/validation file.")
+        else:
+            if self.train_file is not None:
+                extension = self.train_file.split(".")[-1]
+                assert extension in ["csv", "json", "txt"], "`train_file` should be a csv, a json or a txt file."
+            if self.validation_file is not None:
+                extension = self.validation_file.split(".")[-1]
+                assert extension in ["csv", "json", "txt"], "`validation_file` should be a csv, a json or a txt file."
+
+@dataclass
+class FinetuneArguments:
+    """
+    Arguments finetuning with lora config.
+    """
+    lora_rank: int = field(
+        default=8,
+        metadata={
+            "help": "Rank parameter in the LoRA method."
+        },
+    )
+    lora_alpha: int = field(
+        default=32,
+        metadata={
+            "help": "Alpha parameter in the LoRA method."
+        },
+    )
+    lora_dropout: float = field(
+        default=0.1,
+        metadata={
+            "help": "Dropout parameter in the LoRA method."
+        },
+    )
+    lora_target_modules: List[str] = field(
+        default_factory=lambda: ["q", "v"],
+        metadata={
+            "help": "Target modules for the LoRA method."
+        },
+    )
+    peft: Optional[str] = field(
+        default="lora",
+        metadata={
+            "help": (
+                "apply peft. default set to lora"
+            ),
+            "choices": ["lora"],
+        },
+    )
+
+
+def main():
+    # See all possible arguments in src/transformers/training_args.py
+    # or by passing the --help flag to this script.
+    # We now keep distinct sets of args, for a cleaner separation of concerns.
+
+    parser = HfArgumentParser((DataTrainingArguments, ModelArguments, Seq2SeqTrainingArguments, FinetuneArguments))
+    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
+        # If we pass only one argument to the script and it's the path to a json file,
+        # let's parse it to get our arguments.
+        data_args, model_args, training_args, finetune_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
+    else:
+        data_args, model_args, training_args, finetune_args = parser.parse_args_into_dataclasses()
+
+    # Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        handlers=[logging.StreamHandler(sys.stdout)],
+    )
+
+    if training_args.should_log:
+        # The default of training_args.log_level is passive, so we set log level at info here to have that default.
+        transformers.utils.logging.set_verbosity_info()
+
+    log_level = training_args.get_process_log_level()
+    logger.setLevel(log_level)
+    datasets.utils.logging.set_verbosity(log_level)
+    transformers.utils.logging.set_verbosity(log_level)
+    transformers.utils.logging.enable_default_handler()
+    transformers.utils.logging.enable_explicit_format()
+
+    # Log on each process the small summary:
+    logger.warning(
+        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
+        + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
+    )
+    logger.info(f"Training/evaluation parameters {training_args}")
+
+    # Detecting last checkpoint.
+    last_checkpoint = None
+    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
+        last_checkpoint = get_last_checkpoint(training_args.output_dir)
+        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
+            raise ValueError(
+                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
+                "Use --overwrite_output_dir to overcome."
+            )
+        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
+            logger.info(
+                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
+                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
+            )
+
+    # Set seed before initializing model.
+    set_seed(training_args.seed)
+
+    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
+    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
+    # (the dataset will be downloaded automatically from the datasets Hub).
+    #
+    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
+    # 'text' is found. You can easily tweak this behavior (see below).
+    #
+    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
+    # download the dataset.
+    if data_args.dataset_name is not None:
+        # Downloading and loading a dataset from the hub.
+        raw_datasets = load_dataset(
+            data_args.dataset_name,
+            data_args.dataset_config_name,
+            cache_dir=model_args.cache_dir,
+            use_auth_token=True if model_args.use_auth_token else None,
+            streaming=data_args.streaming,
+        )
+
+        if "validation" not in raw_datasets.keys() and training_args.do_eval:
+            raw_datasets["validation"] = load_dataset(
+                data_args.dataset_name,
+                data_args.dataset_config_name,
+                split=f"train[:{data_args.validation_split_percentage}%]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                streaming=data_args.streaming,
+            )
+            raw_datasets["train"] = load_dataset(
+                data_args.dataset_name,
+                data_args.dataset_config_name,
+                split=f"train[{data_args.validation_split_percentage}%:]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                streaming=data_args.streaming,
+            )
+    else:
+        data_files = {}
+        dataset_args = {}
+        if data_args.train_file is not None:
+            data_files["train"] = data_args.train_file
+        if data_args.validation_file is not None:
+            data_files["validation"] = data_args.validation_file
+        extension = (
+            data_args.train_file.split(".")[-1]
+            if data_args.train_file is not None
+            else data_args.validation_file.split(".")[-1]
+        )
+        if extension == "txt":
+            extension = "text"
+            dataset_args["keep_linebreaks"] = data_args.keep_linebreaks
+        raw_datasets = load_dataset(
+            extension,
+            data_files=data_files,
+            cache_dir=model_args.cache_dir,
+            use_auth_token=True if model_args.use_auth_token else None,
+            **dataset_args,
+        )
+
+        # If no validation data is there, validation_split_percentage will be used to divide the dataset.
+        if "validation" not in raw_datasets.keys() and training_args.do_eval:
+            raw_datasets["validation"] = load_dataset(
+                extension,
+                data_files=data_files,
+                split=f"train[:{data_args.validation_split_percentage}%]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                **dataset_args,
+            )
+            raw_datasets["train"] = load_dataset(
+                extension,
+                data_files=data_files,
+                split=f"train[{data_args.validation_split_percentage}%:]",
+                cache_dir=model_args.cache_dir,
+                use_auth_token=True if model_args.use_auth_token else None,
+                **dataset_args,
+            )
+
+
+    tokenizer_kwargs = {
+        "cache_dir": model_args.cache_dir,
+        "use_fast": model_args.use_fast_tokenizer,
+        "revision": model_args.model_revision,
+        "use_auth_token": True if model_args.use_auth_token else None,
+    }
+    if model_args.tokenizer_name:
+        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)
+    elif model_args.model_name_or_path:
+        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
+    else:
+        raise ValueError(
+            "You are instantiating a new tokenizer from scratch. This is not supported by this script."
+            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
+        )
+
+    tokenizer.pad_token = tokenizer.eos_token
+    tokenizer.pad_token_id = tokenizer.eos_token_id
+
+    # Preprocessing the datasets.
+    # First we tokenize all the texts.
+    if training_args.do_train:
+        column_names = list(raw_datasets["train"].features)
+    else:
+        column_names = list(raw_datasets["validation"].features)
+
+    # dataset preprocessing
+    def prepare_features(examples):
+
+        instructions = [q.strip() for q in examples["instruction"]]
+        inputs = [q.strip() for q in examples["input"]]
+
+        responses = [q.strip() for q in examples["output"]]
+
+        examples["input_ids"] = []
+
+        examples["labels"] = []
+
+        for instruction, inpu, response in zip(instructions, inputs, responses):
+            if inpu == "":
+                prompt = PROMPT_DICT["prompt_no_input"].format(instruction)
+            else:
+                prompt = PROMPT_DICT["prompt_input"].format(instruction, inpu)
+
+            history = tokenizer(prompt, max_length=data_args.max_source_length,
+                    truncation=True, add_special_tokens=False)
+
+            gt_resp = tokenizer(response, max_length=data_args.max_target_length, truncation=True)
+
+            input_ids = history.input_ids
+
+            labels = gt_resp.input_ids
+
+            examples["input_ids"].append(input_ids)
+            examples["labels"].append(labels)
+
+        return examples
+
+    if training_args.do_train:
+        if "train" not in raw_datasets:
+            raise ValueError("--do_train requires a train dataset")
+        train_dataset = raw_datasets["train"]
+        if data_args.max_train_samples is not None:
+            # We will select sample from whole data if argument is specified
+            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
+            train_dataset = train_dataset.select(range(max_train_samples))
+        # Create train feature from dataset
+        with training_args.main_process_first(desc="train dataset map pre-processing"):
+            train_dataset = train_dataset.map(
+                prepare_features,
+                batched=True,
+                num_proc=data_args.preprocessing_num_workers,
+                remove_columns=column_names,
+                load_from_cache_file=not data_args.overwrite_cache,
+                desc="Running tokenizer on train dataset",
+            )
+        if data_args.max_train_samples is not None:
+            # Number of samples might increase during Feature Creation, We select only specified max samples
+            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
+            train_dataset = train_dataset.select(range(max_train_samples))
+
+    # Metric
+    metric = evaluate.load("rouge")
+
+    # Define compute metrics function
+    def compute_metrics(eval_preds):
+        preds, labels = eval_preds
+        if isinstance(preds, tuple):
+            preds = preds[0]
+        preds = np.where(preds != -100, preds, tokenizer.pad_token_id)
+        decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
+        # Replace -100 in the labels as we can't decode them.
+        labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
+        decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
+
+        # Some simple post-processing
+        decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)
+
+        result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
+        result = {k: round(v * 100, 4) for k, v in result.items()}
+        prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
+        result["gen_len"] = np.mean(prediction_lens)
+        return result
+
+
+    if training_args.do_eval:
+        if "validation" not in raw_datasets:
+            raise ValueError("--do_eval requires a validation dataset")
+        eval_examples = raw_datasets["validation"]
+        if data_args.max_eval_samples is not None:
+            # We will select sample from whole data
+            max_eval_samples = min(len(eval_examples), data_args.max_eval_samples)
+            eval_examples = eval_examples.select(range(max_eval_samples))
+        # Validation Feature Creation
+        with training_args.main_process_first(desc="validation dataset map pre-processing"):
+            eval_dataset = eval_examples.map(
+                prepare_features,
+                batched=True,
+                num_proc=data_args.preprocessing_num_workers,
+                remove_columns=column_names,
+                load_from_cache_file=not data_args.overwrite_cache,
+                desc="Running tokenizer on validation dataset",
+            )
+        if data_args.max_eval_samples is not None:
+            # During Feature creation dataset samples might increase, we will select required samples again
+            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
+            eval_dataset = eval_dataset.select(range(max_eval_samples))
+
+        def preprocess_logits_for_metrics(logits, labels):
+            if isinstance(logits, tuple):
+                # Depending on the model and config, logits may contain extra tensors,
+                # like past_key_values, but logits always come first
+                logits = logits[0]
+            return logits.argmax(dim=-1)
+    
+    if training_args.do_train:
+
+        # download model & vocab.
+        config_kwargs = {
+            "cache_dir": model_args.cache_dir,
+            "revision": model_args.model_revision,
+            "use_auth_token": True if model_args.use_auth_token else None,
+        }
+        if model_args.config_name:
+            config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)
+        elif model_args.model_name_or_path:
+            config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)
+        else:
+            raise ValueError(
+                "Please provide value for model_name_or_path or config_name."
+            )
+
+        # Load model
+        if model_args.model_name_or_path:
+            model = AutoModelForSeq2SeqLM.from_pretrained(
+                model_args.model_name_or_path,
+                from_tf=bool(".ckpt" in model_args.model_name_or_path),
+                config=config,
+                cache_dir=model_args.cache_dir,
+                revision=model_args.model_revision,
+                use_auth_token=True if model_args.use_auth_token else None,
+            )
+            model.resize_token_embeddings(len(tokenizer))
+        else:
+            raise ValueError("Must provide model_name_or_path to load a pretrained Seq2SeqLM model.")
+
+        # PEFT settings
+        if finetune_args.peft == "lora":
+            peft_config = LoraConfig(
+                r=finetune_args.lora_rank,
+                lora_alpha=finetune_args.lora_alpha,
+                lora_dropout=finetune_args.lora_dropout,
+                target_modules=finetune_args.lora_target_modules,
+                bias="none",
+                task_type=TaskType.SEQ_2_SEQ_LM,
+            )
+
+        # model = prepare_model_for_int8_training(model)
+        model = get_peft_model(model, peft_config)
+        model.print_trainable_parameters()
+
+    if training_args.do_eval and not training_args.do_train:
+        config = PeftConfig.from_pretrained(model_args.model_name_or_path)
+        model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path)
+        model = PeftModel.from_pretrained(model, model_args.model_name_or_path)
+
+    # ignore tokenizer pad token in the loss
+    label_pad_token_id = -100
+    # Data collator
+    data_collator = DataCollatorForSeq2Seq(
+            tokenizer,
+            model=model,
+            label_pad_token_id=label_pad_token_id,
+            pad_to_multiple_of=8)
+
+    # Create Trainer instance
+    trainer = Seq2SeqTrainer(
+            model=model,
+            args=training_args,
+            data_collator=data_collator,
+            train_dataset=train_dataset if training_args.do_train else None,
+            eval_dataset=eval_dataset if training_args.do_eval else None,
+            compute_metrics=compute_metrics,
+            preprocess_logits_for_metrics=preprocess_logits_for_metrics if training_args.do_eval else None,
+            )
+
+   # Training
+    if training_args.do_train:
+        checkpoint = None
+        if training_args.resume_from_checkpoint is not None:
+            checkpoint = training_args.resume_from_checkpoint
+        elif last_checkpoint is not None:
+            checkpoint = last_checkpoint
+        train_result = trainer.train(resume_from_checkpoint=checkpoint)
+
+        model.save_pretrained(training_args.output_dir)
+        tokenizer.save_pretrained(training_args.output_dir)
+
+        metrics = train_result.metrics
+        max_train_samples = (
+            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
+        )
+        metrics["train_samples"] = min(max_train_samples, len(train_dataset))
+
+        trainer.log_metrics("train", metrics)
+        trainer.save_metrics("train", metrics)
+        trainer.save_state()
+
+    # Evaluation
+    if training_args.do_eval:
+        logger.info("*** Evaluate ***")
+        metrics = trainer.evaluate()
+
+        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
+        metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))
+
+        trainer.log_metrics("eval", metrics)
+        trainer.save_metrics("eval", metrics)
+
+
+def _mp_fn(index):
+    # For xla_spawn (TPUs)
+    main()
+
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/.github/workflows/e2eaiok_deltatuner_nightly_pypi.yml b/.github/workflows/e2eaiok_deltatuner_nightly_pypi.yml
new file mode 100644
index 000000000..b17dedbe1
--- /dev/null
+++ b/.github/workflows/e2eaiok_deltatuner_nightly_pypi.yml
@@ -0,0 +1,42 @@
+name: Publish E2EAIOK Deltatuner nightly to PyPI
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  packages: write
+
+jobs:
+  e2eaiok-recdp-nightly-python-pypi:
+    runs-on: self-hosted
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+
+    - name: days since the commit date
+      run: |
+          :
+          timestamp=$(git log --no-walk --date=unix --format=%cd $GITHUB_SHA)
+          days=$(( ( $(date --utc +%s) - $timestamp ) / 86400 ))
+          if [ $days -eq 0 ]; then
+              echo COMMIT_TODAY=true >> $GITHUB_ENV
+          fi
+
+    - name: Build sdist
+      run: |
+        pip install build wheel
+        cd e2eAIOK/deltatuner
+        release_version=$(cat version | head -1)
+        nightly_build_date=`date '+%Y%m%d%H'`
+        nightly_version=${release_version}b${nightly_build_date}
+        echo $nightly_version > version
+        python3 setup.py sdist
+
+    - name: Upload
+      uses: pypa/gh-action-pypi-publish@master
+      with:
+        password: ${{ secrets.PYPI_API_TOKEN_PYRECDP }}
+        packages_dir: e2eAIOK/deltatuner/dist
diff --git a/.github/workflows/unittest.yml b/.github/workflows/unittest_denas.yml
similarity index 81%
rename from .github/workflows/unittest.yml
rename to .github/workflows/unittest_denas.yml
index 6d06c03b6..8d2d1519c 100644
--- a/.github/workflows/unittest.yml
+++ b/.github/workflows/unittest_denas.yml
@@ -1,4 +1,4 @@
-name: Unit Test
+name: Unit Test DENAS
 
 on:
   pull_request:
@@ -9,14 +9,14 @@ on:
     - 'conf/**'
     - 'demo/**'
     - 'Dockerfile-ubuntu/**'
-    - 'e2eAIOK/**'
-    - 'modelzoo/**'
-    - 'tests/unittest/**'
+    - 'e2eAIOK/DeNas/**'
+    - 'e2eAIOK/common/**'
+    - 'tests/unittest/denas/**'
     - 'setup.py'
 
 jobs:
   unit_tests:
-    name: Unit Test
+    name: Unit Test DENAS
     runs-on: self-hosted
     steps:
     - name: Checkout
@@ -45,7 +45,7 @@ jobs:
 
     - name: Run test
       run: |
-        pytest tests/unittest 
+        pytest tests/unittest/denas
 
     - name: After test summary
       run: |
@@ -54,4 +54,3 @@ jobs:
     - name: Submodule package check
       run: |
         python setup.py bdist_wheel --denas && twine check dist/e2eAIOK_denas-*.*.*-py3-none-any.whl && rm -rf build && rm -rf dist
-        python setup.py bdist_wheel --sda && twine check dist/e2eAIOK_sda-*.*.*-py3-none-any.whl && rm -rf build && rm -rf dist 
diff --git a/.github/workflows/unittest_ma.yml b/.github/workflows/unittest_ma.yml
new file mode 100644
index 000000000..156f4117b
--- /dev/null
+++ b/.github/workflows/unittest_ma.yml
@@ -0,0 +1,53 @@
+name: Unit Test MA
+
+on:
+  pull_request:
+    branches:
+    - main
+    paths:
+    - '.github/workflows/**'
+    - 'conf/**'
+    - 'demo/**'
+    - 'Dockerfile-ubuntu/**'
+    - 'e2eAIOK/ModelAdapter/**'
+    - 'e2eAIOK/common/**'
+    - 'modelzoo/**'
+    - 'tests/unittest/ma/**'
+    - 'setup.py'
+
+jobs:
+  unit_tests:
+    name: Unit Test MA
+    runs-on: self-hosted
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.7'
+      
+    - name: Install Dependencies
+      run: |
+        python --version
+        pip install -U pip
+        pip install -U pytest
+        pip install codecov
+        pip install build wheel twine
+        pip install -r e2eAIOK/DeNas/requirements.txt
+        pip install -r e2eAIOK/ModelAdapter/requirements.txt
+        
+    - name: Install package
+      run: |
+        python setup.py sdist && pip install dist/e2eAIOK-*.*.*.tar.gz
+
+    - name: Run test
+      run: |
+        pytest tests/unittest/ma 
+
+    - name: After test summary
+      run: |
+        codecov
diff --git a/.github/workflows/unittest_util.yml b/.github/workflows/unittest_util.yml
new file mode 100644
index 000000000..a02a35393
--- /dev/null
+++ b/.github/workflows/unittest_util.yml
@@ -0,0 +1,53 @@
+name: Unit Test Util
+
+on:
+  pull_request:
+    branches:
+    - main
+    paths:
+    - '.github/workflows/**'
+    - 'Dockerfile-ubuntu/**'
+    - 'e2eAIOK/utils/**'
+    - 'tests/unittest/utils/**'
+    - 'setup.py'
+
+jobs:
+  unit_tests:
+    name: Unit Test Util
+    runs-on: self-hosted
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+      with:
+        submodules: recursive
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.7'
+      
+    - name: Install Dependencies
+      run: |
+        python --version
+        pip install -U pip
+        pip install -U pytest
+        pip install codecov
+        pip install build wheel twine
+        pip install -r e2eAIOK/DeNas/requirements.txt
+        pip install -r e2eAIOK/ModelAdapter/requirements.txt
+        
+    - name: Install package
+      run: |
+        python setup.py sdist && pip install dist/e2eAIOK-*.*.*.tar.gz
+
+    - name: Run test
+      run: |
+        pytest tests/unittest/utils 
+
+    - name: After test summary
+      run: |
+        codecov
+
+    - name: Submodule package check
+      run: |
+        python setup.py bdist_wheel --sda && twine check dist/e2eAIOK_sda-*.*.*-py3-none-any.whl && rm -rf build && rm -rf dist 
diff --git a/e2eAIOK/deltatuner/README.md b/e2eAIOK/deltatuner/README.md
new file mode 100644
index 000000000..18c5665b2
--- /dev/null
+++ b/e2eAIOK/deltatuner/README.md
@@ -0,0 +1,94 @@
+# Deltatuner
+Deltatuner is an extension for [Peft](https://github.com/huggingface/peft) to improve LLM fine-tuning speed through multiple optimizations, including: leverage the compact model constructor [DE-NAS](https://github.com/intel/e2eAIOK/tree/main/e2eAIOK/DeNas) to construct/modify the compact delta layers in a hardware-aware and train-free approach, and adding more new deltatuning algorithms.
+
+## Introduction
+<p align="center">
+  <img width="90%" src="./doc/deltatuner.png">
+</p>
+
+### Key Components
+- Supported parameter efficient finetuning algorithms
+  - [LoRA](https://arxiv.org/pdf/2106.09685.pdf) algorithm
+  - Scaling and Shifting([SSF](https://arxiv.org/abs/2210.08823)) algorithm: Scale and Shift the deep features in a pre-trained model with much less parameters to catch up with the performance of full finetuning
+  - WIP on adding more algos (AdaLora etc.)
+- De-Nas: Automatically construct compact and optimal delta layers with train-free and hardware-aware mode (more details [here](https://community.intel.com/t5/Blogs/Tech-Innovation/Artificial-Intelligence-AI/Multi-Model-Hardware-Aware-Train-Free-Neural-Architecture-Search/post/1479863))
+  - step1: Generate search space for delta layers
+  - step2: Search algorithm populates delta layers for LM
+  - step3: Train-free score evaluates LM with adaptive delta layers
+
+### Features
+- Easy-to-use: provide package install, just need to inject few codes into the original code 
+- Auto-tuning: automatically select best algorithms and delta structure for finetuning model
+
+### Values
+- Saving computation power: reduce the computation power and time required to fine-tune a model by reducing parameter size as well as memory footprint.
+- Improve accuracy: ensure same or no accuracy regression.
+
+## Get Started
+
+### Installation
+- install the python package
+```shell
+pip install deltatuner
+```
+
+### Fast Fine-tuning on Base models
+Below is an example of optimizing [MPT](https://huggingface.co/mosaicml/mpt-7b) model by adding the following few-lines to use the delatuner optimizations. It use the DE-NAS in delatuner to optimize a LLM with LoRA layers to a LLM with compact LoRA layers, so as to efficiently improve the LLM fine-tuning process in peak memory reduction and time speedup. 
+
+```python
+from delta import deltatuner, deltatuner_args
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import LoraConfig, get_peft_model
+from deltatuner import deltatuner, deltatuner_args
+
+# import model from huggingface
+model_id =  "mosaicml/mpt-7b"
+model = AutoModelForCausalLM.from_pretrained(model_id)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+# adding the lora componenents with peft
+config = LoraConfig()
+lora_model = get_peft_model(model, config) 
+# delatuner optimize the model with best lora layer configuration
+deltatuning_args = deltatuner_args.DeltaTunerArguments()
+deltatuner_model = deltatuner.optimize(model=lora_model, tokenizer=tokenizer, deltatuning_args=deltatuning_args)
+...
+```
+
+### API reference
+In above examples, `deltatuner.optimize` is a python function to using deltatuner supported optimization algorithms to optimize the model.
+```python
+def optimize(model, tokenizer, algo: str="auto", deltatuning_args: DeltaTunerArguments=None) -> DeltaTunerModel:
+    '''
+    Parameters:
+        model  - a PreTrainedModel or LoraModel. Specifies what model should be optimized
+        tokenizer - a tokenizer for preprocess text
+        algo (str, optional) – the algorithm. Specifies what type of adapter algorithm (default: “auto”)
+            "auto" – If the input model is mpt, the algorithm is ssf; elif the algorithm is lora
+            "lora" – use the lora algotihm
+            "ssf" – use the ssf algotithm
+        deltatuning_args (optional) – the deltatuner configuration. Specifically, deltatuning_args.denas is to use the denas in the optimization (default: True)
+    Return 
+        DeltaTunerModel - a wrapper of model, which composed of the original properties/function together with adavance properties/function provided by deltatuner
+    '''
+```
+
+
+### Detailed examples
+
+Please refer to [example page](https://github.com/intel/e2eAIOK/tree/main/example) for more use cases on fine-tuning other LLMs with the help of DeltaTuner.
+
+## Model supported matrix
+
+### Causal Language Modeling
+
+| Model        | LoRA | SSF  |
+|--------------| ---- | ---- |
+| GPT-2        |  |  |
+| GPT-J        |  |  |
+| Bloom        |  |  |
+| OPT          |  |  |
+| GPT-Neo      |  |  |
+| Falcon       |  |  |
+| Flan-T5      |  |  |
+| LLaMA        | ✅  | ✅  |
+| MPT          | ✅  | ✅  |
diff --git a/e2eAIOK/deltatuner/deltatuner/__init__.py b/e2eAIOK/deltatuner/deltatuner/__init__.py
new file mode 100644
index 000000000..ec56dde8f
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/__init__.py
@@ -0,0 +1,14 @@
+from .deltatuner_model import DeltaTunerModel
+from .deltatuner import optimize
+from .deltatuner_args import DeltaTunerArguments
+from .mapping import DELTATUNER_TYPE_TO_CONFIG_MAPPING, get_delta_config
+from .tuner import SSFConfig, DeltaSSFModel
+from .utils import (
+    DeltaTunerType, 
+    get_deltatuner_model_state_dict, 
+    set_deltatuner_model_state_dict
+)
+
+from . import scores
+from . import search
+from . import tuner
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/deltatuner.py b/e2eAIOK/deltatuner/deltatuner/deltatuner.py
new file mode 100644
index 000000000..f6a3adbf3
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/deltatuner.py
@@ -0,0 +1,47 @@
+import logging
+from peft import LoraConfig, PeftModel, LoraModel
+from .tuner import SSFConfig
+
+from .deltatuner_model import DeltaTunerModel
+from .deltatuner_args import DeltaTunerArguments
+from .mapping import MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING
+
+logging.basicConfig(level=logging.INFO,
+                format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+logger = logging.getLogger('deltatuner')
+
+SUPPORTED_ALGO = ["auto", "lora", "ssf"]
+
+def optimize(model, tokenizer, algo: str="auto", adapter_name: str="default", deltatuning_args: DeltaTunerArguments=None) -> DeltaTunerModel:
+    if deltatuning_args is None:
+        deltatuning_args = DeltaTunerArguments()
+    if algo not in SUPPORTED_ALGO:
+        raise NotImplementedError("Current algorithm {} is not supported in deltatuner. ".format(algo))
+    
+    if algo == "auto":
+        if "mpt" in model.config.model_type.lower() or not isinstance(model, PeftModel):
+            algo = "ssf"
+        else:
+            algo = "lora"
+
+    if algo == "ssf" and not isinstance(model, PeftModel):
+        peft_config = SSFConfig(target_modules=deltatuning_args.ssf_target_module, bias=None, task_type=deltatuning_args.task_type)
+        peft_config.base_model_name_or_path = model.__dict__.get("name_or_path", None)
+    elif algo == "lora" and isinstance(model, PeftModel) and isinstance(model.peft_config[adapter_name], LoraConfig):
+        peft_config = model.peft_config[adapter_name]
+        if not deltatuning_args.denas:
+            return model
+    else:
+        raise NotImplementedError("Current algorithm {} does not support {} type of model in deltatuner. Please specify the right algo with the type of model.".format(algo, type(model)))
+
+    whether_denas = "enable" if deltatuning_args.denas else "disable"
+    logging.info("The user specifies or automatically adjasts {} algo on model {} with {} denas".format(algo, model.config.model_type, whether_denas))
+
+    if isinstance(peft_config, LoraConfig) or isinstance(peft_config, SSFConfig):
+        if peft_config.task_type not in MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING:
+            model = DeltaTunerModel(model, tokenizer, peft_config, adapter_name, deltatuning_args)
+        else:
+            model = MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING[peft_config.task_type](model, tokenizer, peft_config, adapter_name, deltatuning_args)
+        return model
+    else:
+        raise NotImplementedError("Current algorithm {} is not supported in deltatuner. ".format(algo))
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/deltatuner_args.py b/e2eAIOK/deltatuner/deltatuner/deltatuner_args.py
new file mode 100644
index 000000000..8e0376475
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/deltatuner_args.py
@@ -0,0 +1,34 @@
+from typing import List
+from dataclasses import dataclass, field
+from transformers import AutoTokenizer
+
+@dataclass
+class DeltaTunerArguments:
+    search_engine: str = field(default="EvolutionarySearchEngine", metadata={"help": "Which search strategy used in the DE-NAS. Can be 'EvolutionarySearchEngine', 'RandomSearchEngine' and 'SigoptSearchEngine'"})
+    model_id: str = field(default=None, metadata={"help": "The model id in the hugging face or the local model path"})
+    batch_size: int = field(default=4, metadata={"help": "The batch size used in the search process to calculate the inference"})
+    max_epochs: int = field(default=10, metadata={"help": "The max epochs of the search processing"})
+    scale_factor: int = field(default=10, metadata={"help": "The scaling factor used in the EA search"})
+    select_num: int = field(default=50, metadata={"help": "The num of top candidates stored in the EA search process"})
+    population_num: int = field(default=50, metadata={"help": "The num of population candidates used in the EA/Random search process"})
+    m_prob: float = field(default=0.2, metadata={"help": "The m probability to produce new mutation network depth"})
+    s_prob: float = field(default=0.4, metadata={"help": "The s probability to produce new other mutation network structures"})
+    crossover_num: int = field(default=25, metadata={"help": "The number of crossover in the EA search process"})
+    mutation_num: int = field(default=25, metadata={"help": "The number of mutation in the EA search process"})
+    img_size: int = field(default=32, metadata={"help": "The input or output sequence length of the model."})
+    expressivity_weight: float = field(default=0, metadata={"help": "The weight of expressivity score in the de-score"})
+    complexity_weight: float = field(default=0, metadata={"help": "The weight of complexity score in the de-score"})
+    diversity_weight: float = field(default=1, metadata={"help": "The weight of diversity score in the de-score"})
+    saliency_weight: float = field(default=1, metadata={"help":"The weight of saliency score in the de-score"})
+    latency_weight: float = field(default=0, metadata={"help":"The weight of latency score in the de-score"})
+    max_param_limits: int = field(default=None, metadata={"help": "The max M number of trainable parameter size"})
+    min_param_limits: int = field(default=None, metadata={"help": "The min M number of trainable parameter size"})
+    budget_latency_max: int = field(default=None, metadata={"help": "The max latency limit of the candidate network (etc., '3*10**6' second for your estimation of max latency of LLM forward)"})
+    budget_latency_min: int = field(default=None, metadata={"help": "The min latency limit of the candidate network (etc., '1*10**6' second for your estimation of min latency of LLM forward)"})
+    layer_name: str = field(default="num_hidden_layers", metadata={"help": "The layer name for a specific model search space"})
+    best_model_structure: str = field(default=None, metadata={"help": "The file path to load the best model structure"})
+    search_space_name: List[str] = field(default=None, metadata={"help": "The search space name of delta layers"})
+    random_seed: int = field(default=12345, metadata={"help": "The random seed of the deltatuner search process"})
+    denas: bool = field(default=True, metadata={"help": "Whether to use the denas"})
+    ssf_target_module: List[str] = field(default_factory=lambda: None, metadata={"help": "Target modules for the SSF method."},)
+    task_type: str = field(default="CAUSAL_LM", metadata={"help": "The task type of the deltatuner model"})
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/deltatuner_model.py b/e2eAIOK/deltatuner/deltatuner/deltatuner_model.py
new file mode 100644
index 000000000..8d6baefaa
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/deltatuner_model.py
@@ -0,0 +1,439 @@
+import os
+import torch
+import json
+import inspect
+import random
+import logging
+import numpy as np
+from typing import Any, Optional, Union
+from peft.utils import (
+    PeftType,
+    get_peft_model_state_dict, 
+    set_peft_model_state_dict,
+    WEIGHTS_NAME,
+    SAFETENSORS_WEIGHTS_NAME
+)
+
+from transformers import PreTrainedModel, AutoTokenizer
+from peft import PeftModel, PeftConfig, LoraConfig, AdaLoraConfig
+from peft.utils import PeftType, PromptLearningConfig, WEIGHTS_NAME, SAFETENSORS_WEIGHTS_NAME, hub_file_exists, set_peft_model_state_dict
+from accelerate import dispatch_model, infer_auto_device_map
+from accelerate.utils import get_balanced_memory
+from accelerate.hooks import AlignDevicesHook, remove_hook_from_submodules, add_hook_to_module
+from huggingface_hub import hf_hub_download
+from huggingface_hub.utils import EntryNotFoundError
+from safetensors.torch import load_file as safe_load_file
+from safetensors.torch import save_file as safe_save_file
+
+from .deltatuner_args import DeltaTunerArguments
+from .tuner import DeltaLoraModel, DeltaLoraSearchSpace, DeltaSSFModel, DeltaSSFSearchSpace
+from .search import SearchEngineFactory, Timer
+from .search.utils import network_latency
+from .utils import DeltaTunerType, get_deltatuner_model_state_dict, set_deltatuner_model_state_dict
+from typing import Any, Dict, List, Optional, Union
+
+DELTATUNNER_TO_MODEL_MAPPING = {
+     PeftType.LORA: DeltaLoraModel,
+     DeltaTunerType.SSF: DeltaSSFModel,
+}
+
+DELTATUNNER_TO_SEARCH_SPACE = {
+     PeftType.LORA: DeltaLoraSearchSpace,
+     DeltaTunerType.SSF: DeltaSSFSearchSpace
+}
+
+MODEL_TYPE_TO_LAYER_NAME = {
+    "mpt": "n_layers"
+}
+
+def setup_seed(seed):
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+
+class DeltaTunerModel(PeftModel, torch.nn.Module):
+    def __init__(self, model, tokenizer: AutoTokenizer, peft_config: PeftConfig, adapter_name: str = "default", denas_config: DeltaTunerArguments = None):
+        torch.nn.Module.__init__(self)
+        self.base_model = model
+        self.tokenizer = tokenizer
+        self.config = getattr(self.base_model, "config", {"model_type": "custom"})
+        self.modules_to_save = None
+        self.best_model_param = None
+        self.peft_config = {}
+        self.active_adapter = adapter_name
+        self.peft_type = peft_config.peft_type
+        self.peft_config[adapter_name] = peft_config
+        logging.basicConfig(level=logging.INFO,
+                format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+        self.logger = logging.getLogger('deltatuner')
+        if not isinstance(model, PeftModel):
+            self.base_model = DELTATUNNER_TO_MODEL_MAPPING[peft_config.peft_type](
+                self.base_model, self.peft_config, adapter_name
+            )
+            self.set_additional_trainable_modules(peft_config, adapter_name)
+
+            if getattr(model, "is_gradient_checkpointing", True):
+                model = self._prepare_model_for_gradient_checkpointing(model)
+
+            # the `pretraining_tp` is set for some models to simulate Tensor Parallelism during inference to avoid
+            # numerical differences, https://github.com/pytorch/pytorch/issues/76232 - to avoid any unexpected
+            # behavior we disable that in this line.
+            if hasattr(self.base_model, "config") and hasattr(self.base_model.config, "pretraining_tp"):
+                self.base_model.config.pretraining_tp = 1
+
+        self.denas_config = denas_config
+        if self.denas_config.denas:
+            self._init_denas_params_()
+            search_space, search_space_name = DELTATUNNER_TO_SEARCH_SPACE[peft_config.peft_type].generate_search_space(model, self.denas_config)
+            supernet_config =  DELTATUNNER_TO_SEARCH_SPACE[peft_config.peft_type].generate_supernet(model, self.denas_config, search_space)
+            if peft_config.peft_type in (DeltaTunerType.SSF):
+                super_net = self.base_model
+            else:
+                super_net = DELTATUNNER_TO_MODEL_MAPPING[peft_config.peft_type](model, self.peft_config, adapter_name, supernet_config)
+            if self.denas_config.best_model_structure and os.path.exists(self.denas_config.best_model_structure):
+                with open(self.denas_config.best_model_structure, "r+") as best_model_param_file:
+                    self.best_model_param = json.loads(best_model_param_file.readline().strip())
+            else:
+                self.denas_config.search_space_name = search_space_name
+                self.best_model_param = json.loads(self.search(self.denas_config, super_net, search_space))
+            self.base_model = super_net.set_sample_config(self.best_model_param)
+
+    def _init_denas_params_(self):
+        if self.config.model_type in MODEL_TYPE_TO_LAYER_NAME:
+            self.denas_config.layer_name = MODEL_TYPE_TO_LAYER_NAME[self.config.model_type]
+        self.denas_config.model_id = self.base_model.config._name_or_path
+        self.denas_config.tokenizer = self.tokenizer
+        self.denas_config.max_param_limits = sum(param.numel() for param in self.base_model.parameters() if param.requires_grad) / 10.**6 if self.denas_config.max_param_limits is None else self.denas_config.max_param_limits
+        self.denas_config.budget_latency_max = network_latency(self.base_model, self.tokenizer, batch_size=self.denas_config.batch_size) if self.denas_config.budget_latency_max is not None else self.denas_config.budget_latency_max
+
+    def search(self, denas_config, super_net, search_space):
+        setup_seed(denas_config.random_seed)
+        with Timer("DE-NAS search best model"):
+            searcher = SearchEngineFactory.create_search_engine(params=denas_config, super_net=super_net, search_space=search_space, peft_type=self.peft_type)
+            best_structure=searcher.search()
+        self.logger.info(f"DE-NAS completed, best structure is {best_structure}")
+        return best_structure
+
+    def save_pretrained(
+        self,
+        save_directory: str,
+        safe_serialization: bool = False,
+        selected_adapters: Optional[List[str]] = None,
+        **kwargs: Any,
+    ):
+        r"""
+        This function saves the adapter model and the adapter configuration files to a directory, so that it can be
+        reloaded using the [`LoraModel.from_pretrained`] class method, and also used by the [`LoraModel.push_to_hub`]
+        method.
+
+        Args:
+            save_directory (`str`):
+                Directory where the adapter model and configuration files will be saved (will be created if it does not
+                exist).
+            kwargs (additional keyword arguments, *optional*):
+                Additional keyword arguments passed along to the `push_to_hub` method.
+        """
+        if os.path.isfile(save_directory):
+            raise ValueError(f"Provided path ({save_directory}) should be a directory, not a file")
+
+        if selected_adapters is None:
+            selected_adapters = list(self.peft_config.keys())
+        else:
+            if any(
+                selected_adapter_name not in list(self.peft_config.keys())
+                for selected_adapter_name in selected_adapters
+            ):
+                raise ValueError(
+                    f"You passed an invalid `selected_adapters` arguments, current supported adapter names are"
+                    f" {list(self.peft_config.keys())} - got {selected_adapters}."
+                )
+
+        os.makedirs(save_directory, exist_ok=True)
+        self.create_or_update_model_card(save_directory)
+
+        for adapter_name in selected_adapters:
+            peft_config = self.peft_config[adapter_name]
+            # save only the trainable weights
+            if peft_config.peft_type in (DeltaTunerType.SSF):
+                output_state_dict = get_deltatuner_model_state_dict(
+                self, state_dict=kwargs.get("state_dict", None), adapter_name=adapter_name
+            )
+            else:
+                output_state_dict = get_peft_model_state_dict(
+                    self, state_dict=kwargs.get("state_dict", None), adapter_name=adapter_name
+                )
+            output_dir = os.path.join(save_directory, adapter_name) if adapter_name != "default" else save_directory
+            os.makedirs(output_dir, exist_ok=True)
+
+            if safe_serialization:
+                safe_save_file(
+                    output_state_dict,
+                    os.path.join(output_dir, SAFETENSORS_WEIGHTS_NAME),
+                    metadata={"format": "pt"},
+                )
+            else:
+                torch.save(output_state_dict, os.path.join(output_dir, WEIGHTS_NAME))
+
+            # save the config and change the inference mode to `True`
+            if peft_config.base_model_name_or_path is None:
+                peft_config.base_model_name_or_path = (
+                    self.base_model.__dict__.get("name_or_path", None)
+                    if isinstance(peft_config, PromptLearningConfig)
+                    else self.base_model.model.__dict__.get("name_or_path", None)
+                )
+            inference_mode = peft_config.inference_mode
+            peft_config.inference_mode = True
+
+            if peft_config.task_type is None:
+                # deal with auto mapping
+                base_model_class = self._get_base_model_class(
+                    is_prompt_tuning=isinstance(peft_config, PromptLearningConfig)
+                )
+                parent_library = base_model_class.__module__
+
+                auto_mapping_dict = {
+                    "base_model_class": base_model_class.__name__,
+                    "parent_library": parent_library,
+                }
+            else:
+                auto_mapping_dict = None
+
+            peft_config.save_pretrained(output_dir, auto_mapping_dict=auto_mapping_dict)
+            peft_config.inference_mode = inference_mode
+
+        if self.best_model_param is not None:
+            with open(os.path.join(save_directory, "best_model_structure.txt"), "w+") as fout:
+                fout.write(json.dumps(self.best_model_param))
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        model: PreTrainedModel,
+        model_id: Union[str, os.PathLike],
+        adapter_name: str = "default",
+        is_trainable: bool = False,
+        config: Optional[PeftConfig] = None,
+        **kwargs: Any,
+    ):
+        from peft.mapping import PEFT_TYPE_TO_CONFIG_MAPPING
+        from .mapping import DELTATUNER_TYPE_TO_CONFIG_MAPPING, MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING
+        denas_config = kwargs.pop("denas_config", None)
+
+        # load the config
+        if config is None:
+            peft_type_name = PeftConfig._get_peft_type(
+                    model_id,
+                    subfolder=kwargs.get("subfolder", None),
+                    revision=kwargs.get("revision", None),
+                    cache_dir=kwargs.get("cache_dir", None),
+                    use_auth_token=kwargs.get("use_auth_token", None),
+                )
+            if peft_type_name in ['SSF']:
+                config = DELTATUNER_TYPE_TO_CONFIG_MAPPING[peft_type_name].from_pretrained(model_id, **kwargs)
+            else:
+                config = PEFT_TYPE_TO_CONFIG_MAPPING[peft_type_name].from_pretrained(model_id, **kwargs)
+        elif isinstance(config, PeftConfig):
+            config.inference_mode = not is_trainable
+        else:
+            raise ValueError(f"The input config must be a PeftConfig, got {config.__class__}")
+
+        if (getattr(model, "hf_device_map", None) is not None) and len(
+            set(model.hf_device_map.values()).intersection({"cpu", "disk"})
+        ) > 0:
+            remove_hook_from_submodules(model)
+
+        if isinstance(config, PromptLearningConfig) and is_trainable:
+            raise ValueError("Cannot set a prompt learning adapter to trainable when loading pretrained adapter.")
+        else:
+            config.inference_mode = not is_trainable
+
+        if config.task_type not in MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING.keys():
+            model = cls(model, config, adapter_name, denas_config)
+        else:
+            model = MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING[config.task_type](model, config, adapter_name, denas_config)
+        model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs)
+        return model
+
+    def load_adapter(self, model_id: str, adapter_name: str, is_trainable: bool = False, **kwargs: Any):
+        from peft.mapping import PEFT_TYPE_TO_CONFIG_MAPPING
+        from .mapping import DELTATUNER_TYPE_TO_CONFIG_MAPPING
+
+        hf_hub_download_kwargs, kwargs = self._split_kwargs(kwargs)
+
+        if adapter_name not in self.peft_config:
+            # load the config
+            peft_type_name = PeftConfig._get_peft_type(
+                    model_id,
+                    **hf_hub_download_kwargs,
+                )
+            if peft_type_name in ["SSF"]:
+                peft_config = DELTATUNER_TYPE_TO_CONFIG_MAPPING[peft_type_name].from_pretrained(
+                    model_id,
+                    **hf_hub_download_kwargs,
+                )
+            else:
+                peft_config = PEFT_TYPE_TO_CONFIG_MAPPING[peft_type_name].from_pretrained(
+                    model_id,
+                    **hf_hub_download_kwargs,
+                )
+            if isinstance(peft_config, PromptLearningConfig) and is_trainable:
+                raise ValueError("Cannot set a prompt learning adapter to trainable when loading pretrained adapter.")
+            else:
+                peft_config.inference_mode = not is_trainable
+            self.add_adapter(adapter_name, peft_config)
+
+        # load weights if any
+        path = (
+            os.path.join(model_id, hf_hub_download_kwargs["subfolder"])
+            if hf_hub_download_kwargs.get("subfolder", None) is not None
+            else model_id
+        )
+
+        if os.path.exists(os.path.join(path, SAFETENSORS_WEIGHTS_NAME)):
+            filename = os.path.join(path, SAFETENSORS_WEIGHTS_NAME)
+            use_safetensors = True
+        elif os.path.exists(os.path.join(path, WEIGHTS_NAME)):
+            filename = os.path.join(path, WEIGHTS_NAME)
+            use_safetensors = False
+        else:
+            has_remote_safetensors_file = hub_file_exists(
+                model_id,
+                SAFETENSORS_WEIGHTS_NAME,
+                revision=hf_hub_download_kwargs.get("revision", None),
+                repo_type=hf_hub_download_kwargs.get("repo_type", None),
+            )
+            use_safetensors = has_remote_safetensors_file
+
+            if has_remote_safetensors_file:
+                # Priority 1: load safetensors weights
+                filename = hf_hub_download(
+                    model_id,
+                    SAFETENSORS_WEIGHTS_NAME,
+                    **hf_hub_download_kwargs,
+                )
+            else:
+                try:
+                    filename = hf_hub_download(model_id, WEIGHTS_NAME, **hf_hub_download_kwargs)
+                except EntryNotFoundError:
+                    raise ValueError(
+                        f"Can't find weights for {model_id} in {model_id} or in the Hugging Face Hub. "
+                        f"Please check that the file {WEIGHTS_NAME} or {SAFETENSORS_WEIGHTS_NAME} is present at {model_id}."
+                    )
+
+        if use_safetensors:
+            adapters_weights = safe_load_file(filename, device=torch.device("cuda" if torch.cuda.is_available() else "cpu"))
+        else:
+            adapters_weights = torch.load(filename, map_location=torch.device("cuda" if torch.cuda.is_available() else "cpu"))
+
+        # load the weights into the model
+        if self.peft_config[adapter_name].peft_type in (DeltaTunerType.SSF):
+            load_result = set_deltatuner_model_state_dict(self, adapters_weights, adapter_name=adapter_name)    
+        else:
+            load_result = set_peft_model_state_dict(self, adapters_weights, adapter_name=adapter_name)
+        if (
+            (getattr(self, "hf_device_map", None) is not None)
+            and (len(set(self.hf_device_map.values()).intersection({"cpu", "disk"})) > 0)
+            and len(self.peft_config) == 1
+        ):
+            device_map = kwargs.get("device_map", "auto")
+            max_memory = kwargs.get("max_memory", None)
+            offload_dir = kwargs.get("offload_folder", None)
+            offload_index = kwargs.get("offload_index", None)
+
+            dispatch_model_kwargs = {}
+            # Safety checker for previous `accelerate` versions
+            # `offload_index` was introduced in https://github.com/huggingface/accelerate/pull/873/
+            if "offload_index" in inspect.signature(dispatch_model).parameters:
+                dispatch_model_kwargs["offload_index"] = offload_index
+
+            no_split_module_classes = self._no_split_modules
+
+            if device_map != "sequential":
+                max_memory = get_balanced_memory(
+                    self,
+                    max_memory=max_memory,
+                    no_split_module_classes=no_split_module_classes,
+                    low_zero=(device_map == "balanced_low_0"),
+                )
+            if isinstance(device_map, str):
+                device_map = infer_auto_device_map(
+                    self, max_memory=max_memory, no_split_module_classes=no_split_module_classes
+                )
+            dispatch_model(
+                self,
+                device_map=device_map,
+                offload_dir=offload_dir,
+                **dispatch_model_kwargs,
+            )
+            hook = AlignDevicesHook(io_same_device=True)
+            if isinstance(self.peft_config[adapter_name], PromptLearningConfig):
+                remove_hook_from_submodules(self.prompt_encoder)
+            add_hook_to_module(self.get_base_model(), hook)
+
+        # Set model in evaluation mode to deactivate Dropout modules by default
+        if not is_trainable:
+            self.eval()
+        return load_result
+
+class DelatunerModelForCausalLM(DeltaTunerModel):
+    def __init__(self, model: PeftModel, tokenizer: AutoTokenizer, peft_config: PeftConfig, adapter_name: str = "default", denas_config: DeltaTunerArguments = None):
+        super().__init__(model, tokenizer, peft_config, adapter_name, denas_config)
+        self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
+    
+    def forward(
+        self,
+        input_ids=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        labels=None,
+        output_attentions=None,
+        output_hidden_states=None,
+        return_dict=None,
+        **kwargs,
+    ):
+        peft_config = self.active_peft_config
+        if self.base_model.config.model_type == "mpt":
+            if inputs_embeds is not None:
+                raise AssertionError("forward in MPTForCausalLM does not support inputs_embeds")
+            return self.base_model(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                labels=labels,
+                output_attentions=output_attentions,
+                output_hidden_states=output_hidden_states,
+                return_dict=return_dict,
+                **kwargs,
+            )
+
+        return self.base_model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            labels=labels,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            **kwargs,
+        )
+
+    def generate(self, **kwargs):
+        self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation
+        if hasattr(self.base_model, "model"):
+            self.base_model.model.generation_config = self.generation_config
+        else:
+            self.base_model.generation_config = self.generation_config
+        try:
+            outputs = self.base_model.generate(**kwargs)
+        except:
+            self.base_model.prepare_inputs_for_generation = self.base_model_prepare_inputs_for_generation
+            raise
+        else:
+            self.base_model.prepare_inputs_for_generation = self.base_model_prepare_inputs_for_generation
+            return outputs
+
+    def prepare_inputs_for_generation(self, *args, **kwargs):
+        peft_config = self.active_peft_config
+        model_kwargs = self.base_model_prepare_inputs_for_generation(*args, **kwargs)
+        
+        return model_kwargs
+        
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/mapping.py b/e2eAIOK/deltatuner/deltatuner/mapping.py
new file mode 100644
index 000000000..18f7a5de3
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/mapping.py
@@ -0,0 +1,22 @@
+from .tuner import SSFConfig
+from typing import TYPE_CHECKING, Any, Dict
+from .deltatuner_model import DeltaTunerModel, DelatunerModelForCausalLM
+
+
+DELTATUNER_TYPE_TO_CONFIG_MAPPING = {
+    "SSF": SSFConfig,
+}
+
+MODEL_TYPE_TO_DELTATUNER_MODEL_MAPPING = {
+    "CAUSAL_LM": DelatunerModelForCausalLM
+}
+
+def get_delta_config(config_dict: Dict[str, Any]):
+    """
+    Returns a Delta Tuner config object from a dictionary.
+
+    Args:
+        config_dict (`Dict[str, Any]`): Dictionary containing the configuration parameters.
+    """
+
+    return DELTATUNER_TYPE_TO_CONFIG_MAPPING[config_dict["peft_type"]](**config_dict)
diff --git a/e2eAIOK/deltatuner/deltatuner/scores/__init__.py b/e2eAIOK/deltatuner/deltatuner/scores/__init__.py
new file mode 100644
index 000000000..50c57a203
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/scores/__init__.py
@@ -0,0 +1,1 @@
+from .compute_de_score import do_compute_nas_score, do_compute_nas_score_transformer
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/scores/compute_de_score.py b/e2eAIOK/deltatuner/deltatuner/scores/compute_de_score.py
new file mode 100644
index 000000000..4985a8ad0
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/scores/compute_de_score.py
@@ -0,0 +1,70 @@
+
+
+import os, sys, time, logging
+import torch
+from torch import nn
+import numpy as np
+import gc
+
+import torch
+
+from ..utils import DeltaTunerType
+from ..search.utils import network_latency, input_constructor
+
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+logger = logging.getLogger('DENAS')
+
+def do_compute_nas_score_transformer(model, tokenizer, resolution, batch_size, expressivity_weight=0, complexity_weight=0, diversity_weight=0, saliency_weight=0, latency_weight=0,peft_type=None):
+    if peft_type in (DeltaTunerType.SSF):
+        from .transformer_proxy_ssf import compute_diversity_score, compute_saliency_score
+    else:
+        from .transformer_proxy import network_weight_gaussian_init, compute_diversity_score, compute_saliency_score
+        network_weight_gaussian_init(model)
+    expressivity_score = 0
+    complexity_score = 0
+    model.train()
+    #model.requires_grad_(True)
+    model.zero_grad() 
+    
+    input, _ = input_constructor(batch_size=batch_size, resolution=resolution, tokenizer=tokenizer)
+    output = model(**input)
+    output = output.logits
+    torch.sum(output).backward()
+
+    diversity_score_list = compute_diversity_score(model)
+    diversity_score = 0
+    for grad_abs in diversity_score_list:
+        if len(grad_abs.shape) == 0:
+            diversity_score += float(grad_abs)
+        elif len(grad_abs.shape) == 2:
+            diversity_score += float(torch.mean(torch.sum(grad_abs, dim=[1])))
+        else:
+            raise RuntimeError("diversity_score only support grad shapre of 0 or 2")
+
+    grads_abs_list = compute_saliency_score(model)
+    saliency_score = 0
+    for grad_abs in grads_abs_list:
+        if len(grad_abs.shape) == 4:
+            saliency_score += float(torch.mean(torch.sum(grad_abs, dim=[1,2,3])))
+        elif len(grad_abs.shape) == 2:
+            saliency_score += float(torch.mean(torch.sum(grad_abs, dim=[1])))
+        elif len(grad_abs.shape) == 1:
+            saliency_score += float(torch.mean(grad_abs))
+        else:
+            raise RuntimeError('saliency_score only support grad shape of 1, 2 or 4')
+    
+    latency = 0
+    if latency_weight != 0:
+        latency = network_latency(model=model, tokenizer=tokenizer, batch_size=batch_size)
+    saliency_score_scale = 10.**5 if peft_type in (DeltaTunerType.SSF) else 10.**9
+    score = (expressivity_score*expressivity_weight / 10.**9
+                    + complexity_score*complexity_weight / 10.**9
+                    + diversity_score*diversity_weight / 10.**9
+                    + saliency_score*saliency_weight / saliency_score_scale)
+    logger.info("saliency score:{}, diversity score:{}, latency_score:{}".format(saliency_score/saliency_score_scale, diversity_score/10.**9, latency))
+    nas_score = score/(1 + latency*latency_weight)
+    return nas_score, score, latency
+
+def do_compute_nas_score(model, tokenizer, resolution, batch_size, mixup_gamma, expressivity_weight=0, complexity_weight=0, diversity_weight=0, saliency_weight=0, latency_weight=0,peft_type=None):
+    nas_score, score, latency = do_compute_nas_score_transformer(model, tokenizer, resolution, batch_size, expressivity_weight, complexity_weight, diversity_weight, saliency_weight, latency_weight,peft_type)
+    return nas_score, score, latency
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/scores/transformer_proxy.py b/e2eAIOK/deltatuner/deltatuner/scores/transformer_proxy.py
new file mode 100644
index 000000000..6e9cae148
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/scores/transformer_proxy.py
@@ -0,0 +1,92 @@
+import os, sys, time, math
+import torch
+from torch import nn
+import numpy as np
+import gc
+import torch
+from torchsummary import summary
+from torch.nn.modules import linear
+from transformers import pytorch_utils
+
+from .utils import ATTN_LAYER_STRUCTURE_NAME, LINEAR_LAYER_STRUCTURE_NAME, LINEAR_MODULE_TYPE, LINEAR_LORA_MODULE_TYPE
+
+# TODO: currently specific support for one pure lora layer
+def network_weight_gaussian_init(model):
+    with torch.no_grad():
+        for k, m in model.named_modules():
+            if any(isinstance(m, module_type) for module_type in LINEAR_LORA_MODULE_TYPE):
+                if hasattr(m, "lora_A") and "default" in m.lora_A.keys():
+                    nn.init.kaiming_uniform_(m.lora_A["default"].weight, a=math.sqrt(5), mode='fan_in')
+                    if hasattr(m.lora_A["default"], "bias") and m.lora_A["default"].bias is not None:
+                        nn.init.zeros_(m.lora_A["default"].bias)
+                if hasattr(m, "lora_B") and "default" in m.lora_B.keys():
+                    nn.init.kaiming_uniform_(m.lora_B["default"].weight, a=math.sqrt(5), mode='fan_out')
+                    if hasattr(m.lora_B["default"], "bias") and m.lora_B["default"].bias is not None:
+                        nn.init.zeros_(m.lora_B["default"].bias)
+    return model
+
+# TODO: separate domain specific ops
+def get_linear_layer_metric_array(net, metric):
+    metric_array = []
+    for k, module in net.named_modules():
+        if not any(name in k for name in LINEAR_LAYER_STRUCTURE_NAME):
+            continue
+        else:
+            target = net.get_submodule(k)
+            if any(isinstance(target, module_type) for module_type in LINEAR_MODULE_TYPE):
+                metric_array.append(metric(target))
+    return metric_array
+
+def get_attn_layer_metric_array(net, metric):
+    metric_array = []
+    for k, module in net.named_modules():
+        if not any(name in k for name in ATTN_LAYER_STRUCTURE_NAME):
+            continue
+        else:
+            target = net.get_submodule(k)
+            if any(isinstance(target, module_type) for module_type in LINEAR_MODULE_TYPE):
+                metric_array.append(metric(target))
+    return metric_array
+
+def compute_diversity_score(net):
+    
+    # select the gradients that we want to use for search/prune
+    def nuc_norm(layer):
+        if layer.weight.grad is not None:
+            return torch.norm(layer.weight, p="nuc") * torch.norm(layer.weight.grad, p="nuc")
+        else:
+            return torch.zeros_like(layer.weight)
+
+    diversity_score_list = get_attn_layer_metric_array(net, nuc_norm)
+    return diversity_score_list
+
+def compute_saliency_score(net):
+    # convert params to their abs. Keep sign for converting it back.
+    @torch.no_grad()
+    def linearize(net):
+        signs = {}
+        for name, param in net.state_dict().items():
+            signs[name] = torch.sign(param)
+            #param.abs_()
+        return signs
+    # convert to orig values
+    @torch.no_grad()
+    def nonlinearize(net, signs):
+        for name, param in net.state_dict().items():
+            if 'weight_mask' not in name:
+                param.mul_(signs[name])
+
+    # keep signs of all params
+    #signs = linearize(net)
+    # select the gradients that we want to use for search/prune
+    def synflow(layer):
+        if layer.weight.grad is not None:
+            return torch.abs(layer.weight * layer.weight.grad)
+        else:
+            return torch.zeros_like(layer.weight)
+
+    grads_abs = get_linear_layer_metric_array(net, synflow)
+    grads_abs.extend(get_attn_layer_metric_array(net, synflow))
+    # apply signs of all params
+    #nonlinearize(net, signs)
+    return grads_abs
diff --git a/e2eAIOK/deltatuner/deltatuner/scores/transformer_proxy_ssf.py b/e2eAIOK/deltatuner/deltatuner/scores/transformer_proxy_ssf.py
new file mode 100644
index 000000000..e7a0a580a
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/scores/transformer_proxy_ssf.py
@@ -0,0 +1,76 @@
+import os, sys, time, math
+import torch
+from torch import nn
+import numpy as np
+import gc
+import torch
+from torchsummary import summary
+from torch.nn.modules import linear
+from transformers import pytorch_utils
+
+from .utils import ATTN_LAYER_STRUCTURE_NAME, LINEAR_LAYER_STRUCTURE_NAME
+from ..search.utils import network_latency, input_constructor
+
+# TODO: separate domain specific ops
+def get_linear_layer_metric_array(net, metric):
+    metric_array = []
+    for k, param in net.named_parameters():
+        if not any(name in k for name in LINEAR_LAYER_STRUCTURE_NAME):
+            continue
+        else:
+            if "ssf_" in k:
+                metric_array.append(metric(param))
+    return metric_array
+
+def get_attn_layer_metric_array(net, metric):
+    metric_array = []
+    for k, param in net.named_parameters():
+        if not any(name in k for name in ATTN_LAYER_STRUCTURE_NAME):
+            continue
+        else:
+            if "ssf_" in k:
+                metric_array.append(metric(param))
+    return metric_array
+
+def compute_diversity_score(net):
+    
+    # select the gradients that we want to use for search/prune
+    def nuc_norm(param):
+        if param.grad is not None:
+            return torch.linalg.vector_norm(param) * torch.linalg.vector_norm(param.grad)
+        else:
+            return torch.zeros_like(param)
+
+    diversity_score_list = get_attn_layer_metric_array(net, nuc_norm)
+    return diversity_score_list
+
+def compute_saliency_score(net):
+    # convert params to their abs. Keep sign for converting it back.
+    @torch.no_grad()
+    def linearize(net):
+        signs = {}
+        for name, param in net.state_dict().items():
+            signs[name] = torch.sign(param)
+            #param.abs_()
+        return signs
+    # convert to orig values
+    @torch.no_grad()
+    def nonlinearize(net, signs):
+        for name, param in net.state_dict().items():
+            if 'weight_mask' not in name:
+                param.mul_(signs[name])
+
+    # keep signs of all params
+    #signs = linearize(net)
+    # select the gradients that we want to use for search/prune
+    def synflow(param):
+        if param.grad is not None:
+            return torch.abs(param * param.grad)
+        else:
+            return torch.zeros_like(param)
+
+    grads_abs = get_linear_layer_metric_array(net, synflow)
+    grads_abs.extend(get_attn_layer_metric_array(net, synflow))
+    # apply signs of all params
+    #nonlinearize(net, signs)
+    return grads_abs
diff --git a/e2eAIOK/deltatuner/deltatuner/scores/utils.py b/e2eAIOK/deltatuner/deltatuner/scores/utils.py
new file mode 100644
index 000000000..aec1317af
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/scores/utils.py
@@ -0,0 +1,11 @@
+import torch
+from transformers import pytorch_utils
+from peft.tuners.lora import Linear, Conv1D, LoraLayer
+
+LINEAR_LAYER_STRUCTURE_NAME = ["output", "intermediate", "mlp", "ffn"]
+
+ATTN_LAYER_STRUCTURE_NAME = ["attn"]
+
+LINEAR_MODULE_TYPE =  [torch.nn.Linear, pytorch_utils.Conv1D, Linear, Conv1D]
+
+LINEAR_LORA_MODULE_TYPE = [LoraLayer]
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/search/BaseSearchEngine.py b/e2eAIOK/deltatuner/deltatuner/search/BaseSearchEngine.py
new file mode 100644
index 000000000..718a7e79c
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/search/BaseSearchEngine.py
@@ -0,0 +1,89 @@
+import logging
+import gc
+import json
+import numpy as np
+from transformers import AutoTokenizer
+
+from abc import ABC, abstractmethod
+from .utils import network_latency, network_is_legal, populate_random_func
+from ..scores import do_compute_nas_score
+
+ 
+class BaseSearchEngine(ABC):
+    def __init__(self, params=None, super_net=None, search_space=None, peft_type=None):
+        super().__init__()
+        self.super_net = super_net
+        self.search_space = search_space
+        self.params = params
+        self.peft_type=peft_type
+        if self.params.tokenizer is None:
+            raise RuntimeError("Please specify the right tokenizer in the deltatuner algo!")
+        self.tokenizer =  self.params.tokenizer
+        logging.basicConfig(level=logging.INFO,
+                format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+        self.logger = logging.getLogger('DENAS')
+    
+    @abstractmethod
+    def search(self):
+        pass
+
+    @abstractmethod
+    def get_best_structures(self):
+        pass
+
+    '''
+    Judge sample structure legal or not
+    '''
+    def cand_islegal(self, cand):
+        return network_is_legal(cand, self.vis_dict, self.params, self.super_net)
+
+    '''
+    Check whether candidate latency is within latency budget
+    '''
+    def cand_islegal_latency(self, cand):
+        if hasattr(self.params, "budget_latency_max") or hasattr(self.params, "budget_latency_min"):
+            latency = self.get_latency(cand)
+            if hasattr(self.params, "budget_latency_max") and self.params.budget_latency_max is not None and self.params.budget_latency_max < latency:
+                return False
+            if hasattr(self.params, "budget_latency_min") and self.params.budget_latency_min is not None and self.params.budget_latency_min > latency:
+                return False
+        return True
+
+    '''
+    Compute candidate latency
+    '''
+    def get_latency(self, cand):
+        if 'latency' in self.vis_dict[cand]:
+            return self.vis_dict[cand]['latency']
+        latency = np.inf
+        cand_dict = json.loads(cand)
+        #model = self.super_net.set_sample_config(cand_dict)
+        latency = network_latency(self.super_net, self.tokenizer, batch_size=self.params.batch_size)
+        self.vis_dict[cand]['latency']= latency
+        return self.vis_dict[cand]['latency']
+
+    '''
+    Generate sample random structure
+    '''
+    def populate_random_func(self):
+        return populate_random_func(self.search_space, self.params, self.super_net, self.params.search_space_name)
+
+    '''
+    Compute nas score for sample structure
+    '''
+    def cand_evaluate(self, cand):
+        cand_dict = json.loads(cand)
+        #model = self.super_net.set_sample_config(cand_dict)
+        nas_score, score, latency = do_compute_nas_score(model=self.super_net, 
+                                                        tokenizer=self.tokenizer,
+                                                        resolution=self.params.img_size,
+                                                        batch_size=self.params.batch_size,
+                                                        mixup_gamma=1e-2,
+                                                        expressivity_weight=self.params.expressivity_weight,
+                                                        complexity_weight=self.params.complexity_weight,
+                                                        diversity_weight=self.params.diversity_weight,
+                                                        saliency_weight=self.params.saliency_weight,
+                                                        latency_weight=self.params.latency_weight,
+                                                        peft_type=self.peft_type)
+        self.vis_dict[cand]['score'] = nas_score
+        return nas_score, score, latency
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/search/EvolutionarySearchEngine.py b/e2eAIOK/deltatuner/deltatuner/search/EvolutionarySearchEngine.py
new file mode 100644
index 000000000..67bac46a0
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/search/EvolutionarySearchEngine.py
@@ -0,0 +1,129 @@
+import os
+import sys
+import traceback
+from .BaseSearchEngine import BaseSearchEngine
+from .utils import mutation_random_func, crossover_random_func
+
+class EvolutionarySearchEngine(BaseSearchEngine):
+
+    def __init__(self, params=None, super_net=None, search_space=None,peft_type=None):
+        super().__init__(params,super_net,search_space,peft_type)
+        self.candidates = []
+        self.top_candidates = []
+        self.vis_dict = {}
+
+    '''
+    Higher order function that stack random candidates
+    '''
+    def stack_random_cand(self, random_func, batchsize=10):
+        while True:
+            cands = [random_func() for _ in range(batchsize)]
+            for cand in cands:
+                if cand not in self.vis_dict:
+                    self.vis_dict[cand] = {}
+            for cand in cands:
+                yield cand
+
+    '''
+    EA mutation function for random structure
+    '''
+    def mutation_random_func(self):
+        return mutation_random_func(self.params.m_prob, self.params.s_prob, self.search_space, self.super_net, self.params, self.top_candidates, self.params.search_space_name)
+
+    '''
+    EA crossover function for random structure
+    '''
+    def crossover_random_func(self):
+        return crossover_random_func(self.top_candidates, self.params, self.super_net, self.params.search_space_name)
+
+    '''
+    Supernet decoupled EA populate process
+    '''
+    def get_populate(self):
+        res = []
+        max_iters = self.params.scale_factor * self.params.population_num 
+        cand_iter = self.stack_random_cand(self.populate_random_func)
+        while len(res) < self.params.population_num and max_iters > 0:
+            max_iters -= 1
+            cand = next(cand_iter)
+            if not self.cand_islegal(cand):
+                continue
+            if not self.cand_islegal_latency(cand):
+                continue
+            self.cand_evaluate(cand)
+            res.append(cand)
+            self.logger.info('random {}/{} structure {} nas_score {} params {}'.format(len(res), self.params.population_num, cand, self.vis_dict[cand]['score'], self.vis_dict[cand]['params']))
+        self.logger.info('random_num = {}'.format(len(res)))
+        self.candidates += res
+
+    '''
+    Supernet decoupled EA mutation process
+    '''
+    def get_mutation(self):
+        res = []
+        max_iters = self.params.scale_factor * self.params.mutation_num  
+        cand_iter = self.stack_random_cand(self.mutation_random_func)
+        while len(res) < self.params.mutation_num and max_iters > 0:
+            max_iters -= 1
+            cand = next(cand_iter)
+            if not self.cand_islegal(cand):
+                continue
+            if not self.cand_islegal_latency(cand):
+                continue
+            self.cand_evaluate(cand)
+            res.append(cand)
+            self.logger.info('mutation {}/{} structure {} nas_score {} params {}'.format(len(res), self.params.mutation_num, cand, self.vis_dict[cand]['score'], self.vis_dict[cand]['params']))
+        self.logger.info('mutation_num = {}'.format(len(res)))
+        return res
+
+    '''
+    Supernet decoupled EA crossover process
+    '''
+    def get_crossover(self):
+        res = []
+        max_iters = self.params.scale_factor * self.params.crossover_num
+        cand_iter = self.stack_random_cand(self.crossover_random_func)
+        while len(res) < self.params.crossover_num and max_iters > 0:
+            max_iters -= 1
+            cand = next(cand_iter)
+            if not self.cand_islegal(cand):
+                continue
+            if not self.cand_islegal_latency(cand):
+                continue
+            self.cand_evaluate(cand)
+            res.append(cand)
+            self.logger.info('crossover {}/{} structure {} nas_score {} params {}'.format(len(res), self.params.crossover_num, cand, self.vis_dict[cand]['score'], self.vis_dict[cand]['params']))
+        self.logger.info('crossover_num = {}'.format(len(res)))
+        return res
+
+    '''
+    Keep top candidates of select_num
+    '''
+    def update_population_pool(self):
+        t = self.top_candidates
+        t += self.candidates
+        t.sort(key=lambda x: self.vis_dict[x]['score'], reverse=True)
+        self.top_candidates = t[:self.params.select_num]
+
+    '''
+    Unified API for EvolutionarySearchEngine
+    '''
+    def search(self):
+        for epoch in range(self.params.max_epochs):
+            self.logger.info('epoch = {}'.format(epoch))
+            self.get_populate()
+            self.update_population_pool()
+            mutation = self.get_mutation()
+            crossover = self.get_crossover()
+            self.candidates = mutation + crossover
+            
+        self.update_population_pool()
+        return str(self.top_candidates[0])
+
+    '''
+    Unified API to get best searched structure
+    '''
+    def get_best_structures(self):
+        best_structure = self.top_candidates[0]
+        self.logger.info('best structure {} nas_score {} params {}'.format(best_structure, self.vis_dict[best_structure]['score'], self.vis_dict[best_structure]['params']))
+        return best_structure
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/search/SearchEngineFactory.py b/e2eAIOK/deltatuner/deltatuner/search/SearchEngineFactory.py
new file mode 100644
index 000000000..6725685d1
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/search/SearchEngineFactory.py
@@ -0,0 +1,14 @@
+from .EvolutionarySearchEngine import EvolutionarySearchEngine
+
+SEARCHER_TYPES = {
+        "EvolutionarySearchEngine": EvolutionarySearchEngine
+}
+
+class SearchEngineFactory(object):
+
+    @staticmethod
+    def create_search_engine(params, super_net, search_space,peft_type):
+        if params.search_engine in SEARCHER_TYPES:
+            return SEARCHER_TYPES[params.search_engine](params, super_net, search_space,peft_type)
+        else:
+            raise RuntimeError(f"Search Engine {params.search_engine} is not supported")
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/search/__init__.py b/e2eAIOK/deltatuner/deltatuner/search/__init__.py
new file mode 100644
index 000000000..003d99245
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/search/__init__.py
@@ -0,0 +1,2 @@
+from .SearchEngineFactory import SearchEngineFactory
+from .utils import Timer
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/search/utils.py b/e2eAIOK/deltatuner/deltatuner/search/utils.py
new file mode 100644
index 000000000..b25ee56de
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/search/utils.py
@@ -0,0 +1,117 @@
+import sys
+import os
+import json
+import time
+import timeit
+import yaml
+import random
+import torch
+from transformers import AutoTokenizer
+from transformers.generation.utils import GenerationConfig
+
+
+class Timer:
+    level = 0
+
+    def __init__(self, name):
+        self.name = name
+
+    def __enter__(self):
+        self.start = timeit.default_timer()
+        Timer.level += 1
+
+    def __exit__(self, *a, **kw):
+        Timer.level -= 1
+        print(f'{"  " * Timer.level}{self.name} took {timeit.default_timer() - self.start} sec')
+
+def network_latency(model, tokenizer, batch_size=1, max_seq_length=32, infer_cnt=3.):
+    batch, _ = input_constructor(batch_size, max_seq_length, tokenizer)
+    aver_time = 0.
+    model.eval()
+    for i in range(int(infer_cnt)):
+        start = time.time()
+        with torch.no_grad():
+            _ = model(**batch)
+        end = time.time()
+        sep = 1000 * (end - start)
+        if i == 0:
+            continue
+        else:
+            aver_time += sep / (infer_cnt)
+    return aver_time
+
+def input_constructor(batch_size, resolution, tokenizer):
+    #tokenizer.pad_token = tokenizer.eos_token
+    prompt = ["Once upon a time",
+              "It was in the",
+              "In his lifetime and",
+              "the close links that",
+              "In 2000 Boulter had",
+              "a noted politician and",
+              "He had an elder",
+              "which he was attracted",
+    ]
+    prompt = prompt[:batch_size]
+    inputs = tokenizer(prompt, return_tensors="pt", truncation=True)
+    return inputs, tokenizer
+
+def network_is_legal(cand, vis_dict, params, supernet):
+    if cand not in vis_dict:
+        vis_dict[cand] = {}
+    info = vis_dict[cand]
+    if 'visited' in info:
+        return False
+    cand = json.loads(cand)
+    num_layers = sum(cand["num_hidden_layers"])
+    if num_layers < 1:
+        return False
+    candidate_net = supernet.set_sample_config(cand)
+    n_parameters = sum(param.numel() for param in candidate_net.parameters() if param.requires_grad)
+    info['params'] = n_parameters / 10.**6
+    if params.max_param_limits and info['params'] > params.max_param_limits:
+        return False
+    if params.min_param_limits and info['params'] < params.min_param_limits:
+        return False
+    info['visited'] = True
+    return True
+
+def populate_random_func(search_space, denas_config, supernet, search_space_name):
+    cand_tuple = dict()
+    for name in search_space_name:
+        cand_tuple[name] = []
+    for name in search_space_name:
+        for i in range(getattr(supernet.config, denas_config.layer_name)):
+            cand_tuple[name].append(random.choice(search_space[f"{name}_{i}"]))
+    return json.dumps(cand_tuple)
+
+def mutation_random_func(m_prob, s_prob, search_space, supernet, denas_config, top_candidates, search_space_name):
+    cand = random.choice(top_candidates)
+    result_cand = json.loads(cand)
+    for i in range(getattr(supernet.config, denas_config.layer_name)):
+        random_s = random.random()
+        if random_s < s_prob:
+            result_cand["num_hidden_layers"][i] = random.choice(search_space["num_hidden_layers_{}".format(i)])
+    for name in search_space_name:
+        if name == "num_hidden_layers":
+            continue
+        for i in range(getattr(supernet.config, denas_config.layer_name)):
+            random_s = random.random()
+            if random_s < m_prob:
+                result_cand[name][i] = random.choice(search_space[f"{name}_{i}"])
+    return json.dumps(result_cand)
+
+def crossover_random_func(top_candidates, denas_config, supernet, search_space_name):
+    p1 = json.loads(random.choice(top_candidates))
+    p2 = json.loads(random.choice(top_candidates))
+    max_iters_tmp = 50
+    while p1 != p2 and max_iters_tmp > 0:
+        max_iters_tmp -= 1
+        p1 = json.loads(random.choice(top_candidates))
+        p2 = json.loads(random.choice(top_candidates))
+    cand = {}
+    for name in search_space_name:
+        cand[name] = []
+    for name in search_space_name:
+        for i in range(getattr(supernet.config, denas_config.layer_name)):        
+            cand[name].append(random.choice([p1[name][i], p2[name][i]]))
+    return json.dumps(cand)
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/tuner/__init__.py b/e2eAIOK/deltatuner/deltatuner/tuner/__init__.py
new file mode 100644
index 000000000..2bde1b174
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/tuner/__init__.py
@@ -0,0 +1,3 @@
+from .deltatuner_lora import DeltaLoraModel, DeltaLoraSearchSpace
+from .deltatuner_ssf import DeltaSSFModel, SSFConfig, DeltaSSFSearchSpace
+from .deltatuner_ssf import Linear as SSFLinear
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/tuner/deltatuner_lora.py b/e2eAIOK/deltatuner/deltatuner/tuner/deltatuner_lora.py
new file mode 100644
index 000000000..82ca8acb3
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/tuner/deltatuner_lora.py
@@ -0,0 +1,149 @@
+import torch
+from peft import LoraModel
+from peft.import_utils import is_bnb_available, is_bnb_4bit_available
+from peft.tuners.lora import _get_submodules, mark_only_lora_as_trainable, _freeze_adapter, LoraLayer, Embedding, Conv2d, Linear, Conv1D
+
+if is_bnb_available():
+    import bitsandbytes as bnb
+    from peft.tuners.lora import Linear8bitLt
+    if is_bnb_4bit_available():
+        from peft.tuners.lora import Linear4bit
+
+class DeltaLoraSearchSpace:
+    @classmethod
+    def generate_search_space(cls, model, denas_config,
+                              max_r=12, r_step=1, 
+                              max_alpha=6, alpha_step=1):
+        search_space = {}
+        search_space_name = ["num_hidden_layers", "r", "alpha"]
+        for i in range(getattr(model.config, denas_config.layer_name)):
+            search_space["num_hidden_layers_{}".format(i)] = [0,1]
+            search_space["r_{}".format(i)] = list(range(r_step, max_r+r_step, r_step))
+            search_space["alpha_{}".format(i)] = list(range(alpha_step, max_alpha+alpha_step, alpha_step))
+        return search_space, search_space_name
+    
+    @classmethod
+    def generate_supernet(cls, model, denas_config, search_space):
+        supernet_config = {}
+        supernet_config["num_hidden_layers"] = [search_space["num_hidden_layers_{}".format(i)][-1] for i in range(getattr(model.config, denas_config.layer_name))]
+        supernet_config["r"] = [search_space["r_{}".format(i)][-1] for i in range(getattr(model.config, denas_config.layer_name))]
+        supernet_config["alpha"] = [search_space["alpha_{}".format(i)][-1] for i in range(getattr(model.config, denas_config.layer_name))]
+        return supernet_config
+
+class DeltaLoraModel(LoraModel):
+    def __init__(self, model, config, adapter_name, model_param):
+        super().__init__(model, config, adapter_name)
+        self.model = model
+        self.forward = self.model.forward
+        self.peft_config = config
+        self.model_param = model_param
+        self.adapter_name = adapter_name
+        self.set_sample_config(self.model_param)
+
+    def _create_org_module(self, target):
+        bias = hasattr(target, "bias") and target.bias is not None
+        loaded_in_4bit = getattr(self.model, "is_loaded_in_4bit", False)
+        loaded_in_8bit = getattr(self.model, "is_loaded_in_8bit", False)
+
+        if loaded_in_8bit and isinstance(target, Linear8bitLt):
+            eightbit_kwargs = {}
+            eightbit_kwargs.update(
+                {
+                    "has_fp16_weights": target.state.has_fp16_weights,
+                    "memory_efficient_backward": target.state.memory_efficient_backward,
+                    "threshold": target.state.threshold,
+                    "index": target.index,
+                }
+            )
+            new_module = bnb.nn.Linear8bitLt(input_features=target.in_features, output_features=target.out_features, bias=bias, **eightbit_kwargs)
+        elif loaded_in_4bit and isinstance(target, Linear4bit):
+            fourbit_kwargs = {}
+            fourbit_kwargs.update(
+                {
+                    "compute_dtype": target.compute_dtype,
+                    "compress_statistics": target.weight.compress_statistics,
+                    "quant_type": target.weight.quant_type,
+                }
+            )
+            new_module = bnb.nn.Linear4bit(target.in_features, target.out_features, bias=bias, **fourbit_kwargs)
+        elif isinstance(target, Embedding):
+            new_module = torch.nn.Embedding(num_embeddings=target.num_embeddings, embedding_dim=target.embedding_dim)
+        elif isinstance(target, Conv2d):
+            out_channels, in_channels = target.weight.size()[:2]
+            kernel_size = target.weight.size()[2:]
+            stride = target.stride
+            padding = target.padding
+            new_module = torch.nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride, padding=padding)
+        else:
+            if isinstance(target, Linear):
+                in_features, out_features = target.in_features, target.out_features
+            elif isinstance(target, Conv1D):
+                in_features, out_features = (target.weight.ds_shape if hasattr(target.weight, "ds_shape") else target.weight.shape)
+            elif isinstance(target, torch.nn.Linear):
+                in_features, out_features = target.in_features, target.out_features
+            else:
+                raise ValueError(
+                    f"Target module {target} is not supported. ")
+            new_module = torch.nn.Linear(in_features, out_features, bias=bias)
+        return new_module
+
+    def _replace_module(self, parent_module, child_name, new_module, old_module):
+        setattr(parent_module, child_name, new_module)
+        if old_module.weight.shape != new_module.weight.shape:
+            old_module.weight.data = old_module.weight.data.T
+        new_module.weight = old_module.weight
+        if hasattr(old_module, "bias"):
+            if old_module.bias is not None:
+                new_module.bias = old_module.bias
+
+        if getattr(old_module, "state", None) is not None:
+            new_module.state = old_module.state
+            new_module.to(old_module.weight.device)
+
+        # dispatch to correct device
+        for name, module in new_module.named_modules():
+            if "lora_" in name:
+                module.to(old_module.weight.device)
+            if "ranknum" in name:
+                module.to(old_module.weight.device)
+
+    def set_sample_config(self, model_config):
+        lora_config = self.peft_config[self.adapter_name]
+        self._check_quantization_dependency()
+        is_target_modules_in_base_model = False
+        key_list = [key for key, _ in self.model.named_modules()]
+        
+        for key in key_list:
+            if not self._check_target_module_exists(lora_config, key):
+                continue
+
+            is_target_modules_in_base_model = True
+            parent, target, target_name = _get_submodules(self.model, key)
+            
+            l_num = [int(k) for k in key.split(".") if k.isnumeric()].pop()
+            r = model_config["r"][l_num]
+            alpha = model_config["alpha"][l_num] * r
+            if model_config["num_hidden_layers"][l_num]:
+                lora_config.r = r
+                lora_config.lora_alpha = alpha
+                if isinstance(target, LoraLayer) and isinstance(target, torch.nn.Conv2d):
+                    target.update_layer_conv2d(self.adapter_name, lora_config.r, lora_config.lora_alpha, lora_config.lora_dropout, lora_config.init_lora_weights,)
+                elif isinstance(target, LoraLayer) and isinstance(target, torch.nn.Embedding):
+                    target.update_layer_embedding(self.adapter_name, lora_config.r, lora_config.lora_alpha, lora_config.lora_dropout, lora_config.init_lora_weights,)
+                elif isinstance(target, LoraLayer):
+                    target.update_layer(self.adapter_name, lora_config.r, lora_config.lora_alpha, lora_config.lora_dropout, lora_config.init_lora_weights,)
+                else:
+                    new_module = self._create_new_module(lora_config, self.adapter_name, target)
+                    self._replace_module(parent, target_name, new_module, target)
+            else:
+                new_module = self._create_org_module(target)
+                self._replace_module(parent, target_name, new_module, target)
+
+        if not is_target_modules_in_base_model:
+            raise ValueError(
+                f"Target modules {lora_config.target_modules} not found in the base model. "
+                f"Please check the target modules and try again.")
+        mark_only_lora_as_trainable(self.model, self.peft_config[self.adapter_name].bias)
+        if self.peft_config[self.adapter_name].inference_mode:
+            _freeze_adapter(self.model, self.adapter_name)
+        return self.model
diff --git a/e2eAIOK/deltatuner/deltatuner/tuner/deltatuner_ssf.py b/e2eAIOK/deltatuner/deltatuner/tuner/deltatuner_ssf.py
new file mode 100644
index 000000000..48cff341b
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/tuner/deltatuner_ssf.py
@@ -0,0 +1,429 @@
+import math
+import re
+import warnings
+from dataclasses import asdict, dataclass, field, replace
+from enum import Enum
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from tqdm import tqdm
+from transformers.pytorch_utils import Conv1D
+
+# from peft.import_utils import is_bnb_available, is_bnb_4bit_available
+from peft.utils import (
+    CLAMP_QUANTILE,
+    COMMON_LAYERS_PATTERN,
+    ModulesToSaveWrapper,
+    PeftConfig,
+    _freeze_adapter,
+    _get_submodules,
+    transpose,
+)
+
+from ..utils import DeltaTunerType, TRANSFORMERS_MODELS_TO_SSF_TARGET_MODULES_MAPPING
+
+# if is_bnb_available():
+#     import bitsandbytes as bnb
+
+@dataclass
+class SSFConfig(PeftConfig):
+    """
+    This is the configuration class to store the configuration of a [`SSFModel`].
+
+    Args:
+        target_modules (`Union[List[str],str]`): The names of the modules to apply ssf to.
+        bias (`str`): Bias type for ssf. Can be 'none', 'all'. If 'all', the
+            corresponding biases will be updated during training. Be aware that this means that, even when disabling
+            the adapters, the model will not produce the same output as the base model would have without adaptation.
+        layers_to_transform (`Union[List[int],int]`):
+            The layer indexes to transform, if this argument is specified, it will apply the ssf transformations on
+            the layer indexes that are specified in this list. If a single integer is passed, it will apply the ssf
+            transformations on the layer at this index.
+        layers_pattern (`str`):
+            The layer pattern name, used only if `layers_to_transform` is different from `None` and if the layer
+            pattern is not in the common layers pattern.
+        modules_to_save (`List[str]`):List of modules apart from SSF layers to be set as trainable
+            and saved in the final checkpoint.
+    """
+
+    bias: str = field(default="none", metadata={"help": "Bias type for SSF. Can be 'none', 'all'"})
+    target_modules: Optional[Union[List[str], str]] = field(
+        default=None,
+        metadata={
+            "help": "List of module names or regex expression of the module names to replace with ssf."
+            "For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$' "
+        },
+    )
+    layers_to_transform: Optional[Union[List, int]] = field(
+        default=None,
+        metadata={
+            "help": "The layer indexes to transform, is this argument is specified, PEFT will transform only the layers indexes that are specified inside this list. If a single integer is passed, PEFT will transform only the layer at this index."
+        },
+    )
+    layers_pattern: Optional[str] = field(
+        default=None,
+        metadata={
+            "help": "The layer pattern name, used only if `layers_to_transform` is different to None and if the layer pattern is not in the common layers pattern."
+        },
+    )
+    modules_to_save: Optional[List[str]] = field(
+        default=None,
+        metadata={
+            "help": "List of modules apart from SSF layers to be set as trainable and saved in the final checkpoint. "
+            "For example, in Sequence Classification or Token Classification tasks, "
+            "the final layer `classifier/score` are randomly initialized and as such need to be trainable and saved."
+        },
+    )
+
+    def __post_init__(self):
+        self.peft_type = DeltaTunerType.SSF
+
+class DeltaSSFSearchSpace:
+    @classmethod
+    def generate_search_space(cls, model, denas_config):
+        search_space = {}
+        search_space_name = ["num_hidden_layers"]
+        for i in range(getattr(model.config, denas_config.layer_name)):
+            search_space["num_hidden_layers_{}".format(i)] = [0,1]
+        return search_space, search_space_name
+    
+    @classmethod
+    def generate_supernet(cls, model, denas_config, search_space):
+        supernet_config = {}
+        supernet_config["num_hidden_layers"] = [search_space["num_hidden_layers_{}".format(i)][-1] for i in range(getattr(model.config, denas_config.layer_name))]
+        return supernet_config
+
+class DeltaSSFModel(torch.nn.Module):
+    """
+    Creates SSF model from a pretrained transformers model.
+
+    Args:
+        model ([`~transformers.PreTrainedModel`]): The model to be adapted.
+        config ([`SSFConfig`]): The configuration of the SSF model.
+
+    Returns:
+        `torch.nn.Module`: The SSF model.
+
+    **Attributes**:
+        - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted.
+        - **peft_config** ([`SSFConfig`]): The configuration of the SSF model.
+    """
+
+    def __init__(self, model, config, adapter_name):
+        super().__init__()
+        self.model = model
+        self.forward = self.model.forward
+        self.peft_config = config
+        self.adapter_name = adapter_name
+        self.add_adapter(adapter_name, self.peft_config[adapter_name])
+
+        # transformers models have a .config attribute, whose presence is assumed later on
+        if not hasattr(self, "config"):
+            self.config = {"model_type": "custom"}
+
+    def add_adapter(self, adapter_name, config=None):
+        if config is not None:
+            model_config = getattr(self.model, "config", {"model_type": "custom"})
+            if hasattr(model_config, "to_dict"):
+                model_config = model_config.to_dict()
+
+            config = self._prepare_ssf_config(config, model_config)
+            self.peft_config[adapter_name] = config
+        self._find_and_replace(adapter_name)
+        mark_only_ssf_as_trainable(self.model, self.peft_config[adapter_name].bias)
+        if self.peft_config[adapter_name].inference_mode:
+            # _freeze_adapter(self.model, adapter_name)
+            for n, p in self.model.named_parameters():
+                p.requires_grad = False
+
+    # def _check_quantization_dependency(self):
+    #     loaded_in_4bit = getattr(self.model, "is_loaded_in_4bit", False)
+    #     loaded_in_8bit = getattr(self.model, "is_loaded_in_8bit", False)
+    #     if (loaded_in_4bit or loaded_in_8bit) and not is_bnb_available():
+    #         raise ImportError(
+    #             "To use SSF with 8-bit or 4-bit quantization, please install the `bitsandbytes` package. "
+    #             "You can install it with `pip install bitsandbytes`."
+    #         )
+
+    def _check_target_module_exists(self, ssf_config, key):
+        if isinstance(ssf_config.target_modules, str):
+            target_module_found = re.fullmatch(ssf_config.target_modules, key)
+        else:
+            target_module_found = any(key.endswith(target_key) for target_key in ssf_config.target_modules)
+            is_using_layer_indexes = getattr(ssf_config, "layers_to_transform", None) is not None
+            layer_indexing_pattern = getattr(ssf_config, "layers_pattern", None)
+            if is_using_layer_indexes and target_module_found:
+                layers_pattern = COMMON_LAYERS_PATTERN if layer_indexing_pattern is None else layer_indexing_pattern
+                layers_pattern = [layers_pattern] if isinstance(layers_pattern, str) else layers_pattern
+
+                for pattern in layers_pattern:
+                    layer_index = re.match(f".*.{pattern}\.(\d+)\.*", key)
+                    if layer_index is not None:
+                        layer_index = int(layer_index.group(1))
+                        if isinstance(ssf_config.layers_to_transform, int):
+                            target_module_found = layer_index == ssf_config.layers_to_transform
+                        else:
+                            target_module_found = layer_index in ssf_config.layers_to_transform
+
+                        break
+                    else:
+                        target_module_found = False
+        return target_module_found
+
+    def _create_new_module(self, ssf_config, adapter_name, target):
+        if isinstance(target, torch.nn.Linear):
+            in_features, out_features = target.in_features, target.out_features
+        else:
+            raise ValueError(
+                f"Target module {target} is not supported. "
+                f"Currently, only `torch.nn.Linear` are supported."
+            )
+        new_module = Linear(target, out_features)
+
+        return new_module
+
+    def _find_and_replace(self, adapter_name):
+        ssf_config = self.peft_config[adapter_name]
+        # self._check_quantization_dependency()
+        is_target_modules_in_base_model = False
+        key_list = [key for key, _ in self.model.named_modules()]
+
+        for key in key_list:
+            if not self._check_target_module_exists(ssf_config, key):
+                continue
+
+            is_target_modules_in_base_model = True
+            parent, target, target_name = _get_submodules(self.model, key)
+
+            new_module = self._create_new_module(ssf_config, adapter_name, target)
+            self._replace_module(parent, target_name, new_module, target)
+
+        if not is_target_modules_in_base_model:
+            raise ValueError(
+                f"Target modules {ssf_config.target_modules} not found in the base model. "
+                f"Please check the target modules and try again."
+            )
+
+    def _replace_module(self, parent_module, child_name, new_module, old_module):
+        setattr(parent_module, child_name, new_module)
+        
+        # dispatch to correct device
+        if hasattr(old_module, "weight"):
+            for name, module in new_module.named_modules():
+                module.to(old_module.weight.device)
+
+    def _create_org_module(self, target):
+        if isinstance(target, Linear):
+            new_module = target.get_layer(merge=False)
+        elif isinstance(target, torch.nn.Linear):
+            new_module = target
+        else:
+            raise ValueError(
+                f"Target module {target} is not supported. ")
+        return new_module
+
+    def set_sample_config(self, model_config):
+        ssf_config = self.peft_config[self.adapter_name]
+        # self._check_quantization_dependency()
+        is_target_modules_in_base_model = False
+        key_list = [key for key, _ in self.model.named_modules()]
+        
+        for key in key_list:
+            if not self._check_target_module_exists(ssf_config, key):
+                continue
+
+            is_target_modules_in_base_model = True
+            parent, target, target_name = _get_submodules(self.model, key)
+            
+            l_num = [int(k) for k in key.split(".") if k.isnumeric()].pop()
+            if model_config["num_hidden_layers"][l_num]:
+                if not isinstance(target, SSFLayer):
+                    new_module = self._create_new_module(ssf_config, self.adapter_name, target)
+                    self._replace_module(parent, target_name, new_module, target)
+            else:
+                new_module = self._create_org_module(target)
+                self._replace_module(parent, target_name, new_module, target)
+
+        if not is_target_modules_in_base_model:
+            raise ValueError(
+                f"Target modules {ssf_config.target_modules} not found in the base model. "
+                f"Please check the target modules and try again.")
+        mark_only_ssf_as_trainable(self.model, self.peft_config[self.adapter_name].bias)
+        if self.peft_config[self.adapter_name].inference_mode:
+            # _freeze_adapter(self.model, self.adapter_name)
+             for n, p in self.model.named_parameters():
+                p.requires_grad = False
+        return self
+
+    def merge_adapter(self):
+        """
+        This method merges the SSF layers into the base model.
+        """
+        for module in self.model.modules():
+            if isinstance(module, SSFLayer):
+                module.merge()
+
+    def unmerge_adapter(self):
+        """
+        This method unmerges the SSF layers from the base model.
+        """
+        for module in self.model.modules():
+            if isinstance(module, SSFLayer):
+                module.unmerge()
+
+    def __getattr__(self, name: str):
+        """Forward missing attributes to the wrapped module."""
+        try:
+            return super().__getattr__(name)  # defer to nn.Module's logic
+        except AttributeError:
+            return getattr(self.model, name)
+
+    def get_peft_config_as_dict(self, inference: bool = False):
+        config_dict = {}
+        for key, value in self.peft_config.items():
+            config = {k: v.value if isinstance(v, Enum) else v for k, v in asdict(value).items()}
+            if inference:
+                config["inference_mode"] = True
+        config_dict[key] = config
+        return config
+
+    @staticmethod
+    def _prepare_ssf_config(peft_config, model_config):
+        if peft_config.target_modules is None:
+            if model_config["model_type"] not in TRANSFORMERS_MODELS_TO_SSF_TARGET_MODULES_MAPPING:
+                raise ValueError("Please specify `target_modules` in `peft_config`")
+            peft_config.target_modules = TRANSFORMERS_MODELS_TO_SSF_TARGET_MODULES_MAPPING[model_config["model_type"]]
+        return peft_config
+
+    def _unload_and_optionally_merge(self, merge=True, progressbar: bool = False):
+        key_list = [key for key, _ in self.model.named_modules() if "ssf" not in key]
+        desc = "Unloading " + ("and merging " if merge else "") + "model"
+        for key in tqdm(key_list, disable=not progressbar, desc=desc):
+            try:
+                parent, target, target_name = _get_submodules(self.model, key)
+            except AttributeError:
+                continue
+            if isinstance(target, SSFLayer):
+                if isinstance(target, Linear):
+                    new_module = target.get_layer(merge=merge)
+                else:
+                    raise NotImplementedError
+                self._replace_module(parent, target_name, new_module, target)
+
+            # save any additional trainable modules part of `modules_to_save`
+            # if isinstance(target, ModulesToSaveWrapper):
+            #     setattr(parent, target_name, target.modules_to_save[target.active_adapter])
+
+        return self.model
+
+    def merge_and_unload(self, progressbar: bool = False):
+        r"""
+        This method merges the SSF layers into the base model. This is needed if someone wants to use the base model
+        as a standalone model.
+
+        Args:
+            progressbar (bool): whether to show a progressbar indicating the unload and merge process
+        """
+        return self._unload_and_optionally_merge(progressbar=progressbar)
+
+    def unload(self):
+        """
+        Gets back the base model by removing all the ssf modules without merging. This gives back the original base
+        model.
+        """
+        return self._unload_and_optionally_merge(merge=False)
+
+# had to adapt it for `ssf_only` to work
+def mark_only_ssf_as_trainable(model: nn.Module, bias: str = "none") -> None:
+    for n, p in model.named_parameters():
+        if "ssf_" not in n:
+            p.requires_grad = False
+    if bias is None or bias == "none":
+        return
+    elif bias == "all":
+        for n, p in model.named_parameters():
+            if "bias" in n:
+                p.requires_grad = True
+    else:
+        raise NotImplementedError
+
+class SSFLayer:
+    def __init__(self, dim: int, **kwargs):
+        self.dim = dim
+        
+        self.ssf_scale = nn.Parameter(torch.ones(dim))
+        self.ssf_shift = nn.Parameter(torch.zeros(dim))
+        
+        nn.init.normal_(self.ssf_scale, mean=1, std=.02)
+        nn.init.normal_(self.ssf_shift, std=.02)
+
+    def ssf_ada(self, x):
+        assert self.ssf_scale.shape == self.ssf_shift.shape
+        if x.shape[-1] == self.ssf_scale.shape[0]:
+            return x * self.ssf_scale + self.ssf_shift
+        # elif x.shape[1] == self.ssf_scale.shape[0]:
+        #     return x * self.ssf_scale.view(1, -1, 1, 1) + self.ssf_shift.view(1, -1, 1, 1)
+        else:
+            raise ValueError('the input tensor shape does not match the shape of the scale factor.')
+
+class Linear(nn.Module, SSFLayer):
+    # SSF implemented in a dense layer
+    def __init__(
+        self,
+        target,
+        out_features: int,
+    ):
+        super().__init__()
+        SSFLayer.__init__(self, dim = out_features)
+        self.linear_sub = target
+        self.merged = False
+        
+    def merge(self):
+        if self.merged:
+            warnings.warn("Already merged. Nothing to do.")
+            return
+        
+        #save unmerged data
+        self.linear_sub_weight = self.linear_sub.weight.data.clone()
+        self.linear_sub_bias = None if self.linear_sub.bias is None else self.linear_sub.bias.data.clone()
+
+        # do merge operation
+        self.linear_sub.weight.data = (self.linear_sub.weight.t() * self.ssf_scale).t()
+        if self.linear_sub.bias is None:
+            self.linear_sub.bias = self.ssf_shift
+        else:
+            self.linear_sub.bias.data = self.linear_sub.bias * self.ssf_scale + self.ssf_shift
+        self.merged = True
+        # print("Merge SSF layer into base model")
+    
+    def get_layer(self,merge=True):
+        if merge and not self.merged:
+            self.merge()
+        if not merge and self.merged:
+            self.unmerge()
+        return self.linear_sub
+    
+    def unmerge(self):
+        if not self.merged:
+            warnings.warn("Already unmerged. Nothing to do.")
+            return
+
+        self.linear_sub.weight.data = self.linear_sub_weight
+        if self.linear_sub_bias is None:
+            self.linear_sub.bias = None
+        else:
+            self.linear_sub.bias.data = self.linear_sub_bias
+        self.merged = False
+        # print("Unmerge SSF layer from base model")
+
+    def forward(self, x: torch.Tensor):
+        previous_dtype = x.dtype
+        if self.merged:
+            result = self.linear_sub(x)
+        else:
+            x = self.linear_sub(x)
+            result = self.ssf_ada(x)
+        result = result.to(previous_dtype)
+        return result
diff --git a/e2eAIOK/deltatuner/deltatuner/utils/__init__.py b/e2eAIOK/deltatuner/deltatuner/utils/__init__.py
new file mode 100644
index 000000000..a31b2ae49
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/utils/__init__.py
@@ -0,0 +1,2 @@
+from .config import DeltaTunerType, TRANSFORMERS_MODELS_TO_SSF_TARGET_MODULES_MAPPING
+from .save_and_load import get_deltatuner_model_state_dict, set_deltatuner_model_state_dict
diff --git a/e2eAIOK/deltatuner/deltatuner/utils/config.py b/e2eAIOK/deltatuner/deltatuner/utils/config.py
new file mode 100644
index 000000000..1b3fea166
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/utils/config.py
@@ -0,0 +1,11 @@
+import enum
+from dataclasses import  field
+from typing import List, Optional, Tuple, Union
+
+class DeltaTunerType(str, enum.Enum):
+    SSF = "SSF"
+
+TRANSFORMERS_MODELS_TO_SSF_TARGET_MODULES_MAPPING = {
+    "llama": ["q_proj", "v_proj"],
+    "mpt": ["Wqkv","out_proj","up_proj","down_proj"]
+}
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/deltatuner/utils/save_and_load.py b/e2eAIOK/deltatuner/deltatuner/utils/save_and_load.py
new file mode 100644
index 000000000..35152cfd8
--- /dev/null
+++ b/e2eAIOK/deltatuner/deltatuner/utils/save_and_load.py
@@ -0,0 +1,81 @@
+# coding=utf-8
+# Copyright 2023-present the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from .config import DeltaTunerType
+
+def get_deltatuner_model_state_dict(model, state_dict=None, adapter_name="default"):
+    """
+    Get the state dict of the DeltaTuner model.
+
+    Args:
+        model ([`DeltaTunerModelModel`]): The DeltaTunerModel model. When using torch.nn.DistributedDataParallel, DeepSpeed or FSDP,
+        the model should be the underlying model/unwrapped model (i.e. model.module).
+        state_dict (`dict`, *optional*, defaults to `None`):
+            The state dict of the model. If not provided, the state dict of the model
+        will be used.
+    """
+    config = model.peft_config[adapter_name]
+    if state_dict is None:
+        state_dict = model.state_dict()
+    
+    if config.peft_type == DeltaTunerType.SSF:
+        bias = config.bias
+        if bias is None or bias == "none":
+            to_return = {k: state_dict[k] for k in state_dict if "ssf_" in k}
+        elif bias == "all":
+            to_return = {k: state_dict[k] for k in state_dict if "ssf_" in k or "bias" in k}
+        else:
+            raise NotImplementedError
+    else:
+        raise NotImplementedError
+    if model.modules_to_save is not None:
+        for key, value in state_dict.items():
+            if any(f"{module_name}.modules_to_save.{adapter_name}" in key for module_name in model.modules_to_save):
+                to_return[key.replace("modules_to_save.", "")] = value
+
+    to_return = {k.replace(f".{adapter_name}", ""): v for k, v in to_return.items()}
+    return to_return
+
+
+def set_deltatuner_model_state_dict(model, peft_model_state_dict, adapter_name="default"):
+    """
+    Set the state dict of the DeltaTuner model.
+
+    Args:
+        model ([`DeltaTunerModel`]): The DeltaTuner model.
+        peft_model_state_dict (`dict`): The state dict of the DeltaTuner model.
+    """
+    config = model.peft_config[adapter_name]
+    state_dict = {}
+    if model.modules_to_save is not None:
+        for key, value in peft_model_state_dict.items():
+            if any(module_name in key for module_name in model.modules_to_save):
+                for module_name in model.modules_to_save:
+                    if module_name in key:
+                        key = key.replace(module_name, f"{module_name}.modules_to_save.{adapter_name}")
+                        break
+            state_dict[key] = value
+    else:
+        state_dict = peft_model_state_dict
+
+    if config.peft_type in (DeltaTunerType.SSF):
+        peft_model_state_dict = {}
+        for k, v in state_dict.items():
+            peft_model_state_dict[k] = v
+    else:
+        raise NotImplementedError
+
+    load_result = model.load_state_dict(peft_model_state_dict, strict=False)
+    return load_result
diff --git a/e2eAIOK/deltatuner/doc/deltatuner.png b/e2eAIOK/deltatuner/doc/deltatuner.png
new file mode 100644
index 000000000..dcb660e79
Binary files /dev/null and b/e2eAIOK/deltatuner/doc/deltatuner.png differ
diff --git a/e2eAIOK/deltatuner/docker/Dockerfile b/e2eAIOK/deltatuner/docker/Dockerfile
new file mode 100644
index 000000000..59ae92cc9
--- /dev/null
+++ b/e2eAIOK/deltatuner/docker/Dockerfile
@@ -0,0 +1,58 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+#
+# ============================================================================
+# How to build: 
+#   docker build ./ -f Dockerfile -t chatbot_finetune:latest
+# If you need to use proxy, please use the following command
+#   docker build ./ --build-arg http_proxy=${http_proxy} --build-arg https_proxy=${http_proxy} -f Dockerfile -t chatbot_finetune:latest
+
+FROM ubuntu:22.04
+
+# See http://bugs.python.org/issue19846
+ENV LANG C.UTF-8
+
+# Install system dependencies
+RUN apt-get update \
+    && apt-get install -y build-essential \
+    && apt-get install -y wget numactl git \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install miniconda
+ENV CONDA_DIR /opt/conda
+RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
+    /bin/bash ~/miniconda.sh -b -p /opt/conda
+
+# Put conda in path so we can use conda activate
+SHELL ["/bin/bash", "--login", "-c"]
+ENV PATH=$CONDA_DIR/bin:$PATH
+RUN conda init bash && \
+    unset -f conda && \
+    export PATH=$CONDA_DIR/bin/:${PATH} && \
+    conda config --add channels intel && \
+    conda create -yn chatbot-finetuning python=3.9 && \
+    echo "conda activate chatbot-finetuning" >> ~/.bashrc && \
+    source ~/.bashrc && \
+    pip install torch==1.13.0 transformers datasets accelerate SentencePiece \
+    evaluate peft==0.4.0 torchsummary \
+    nltk rouge_score protobuf==3.20.1 tokenizers einops wandb \
+    sigopt
diff --git a/e2eAIOK/deltatuner/docker/DockerfileGPU b/e2eAIOK/deltatuner/docker/DockerfileGPU
new file mode 100644
index 000000000..60cab9697
--- /dev/null
+++ b/e2eAIOK/deltatuner/docker/DockerfileGPU
@@ -0,0 +1,51 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+#
+# ============================================================================
+# How to build: 
+#  docker build -f docker/DockerfileGPU -t chatbot_finetune:latest .
+
+FROM pytorch/pytorch:latest
+
+# Install system dependencies
+RUN apt-get update \
+    && apt-get install -y build-essential \
+    && apt-get install -y wget numactl git \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Put conda in path so we can use conda activate
+SHELL ["/bin/bash", "--login", "-c"]
+RUN conda init bash && \
+    unset -f conda && \
+    echo "conda activate base" >> ~/.bashrc && \
+    source ~/.bashrc && \
+    pip install transformers datasets accelerate SentencePiece \
+    peft==0.4.0 torchsummary sigopt evaluate \
+    nltk rouge_score protobuf==3.20.1 tokenizers einops \
+    wandb
+
+RUN git clone https://github.com/EleutherAI/lm-evaluation-harness.git ~/lm-evaluation-harness && \
+    cd ~/lm-evaluation-harness && \
+    git checkout b281b0921b636bc3 && \
+    pip install -e . 
+
+RUN conda clean --all -y
\ No newline at end of file
diff --git a/e2eAIOK/deltatuner/setup.py b/e2eAIOK/deltatuner/setup.py
new file mode 100644
index 000000000..1d4fad3e7
--- /dev/null
+++ b/e2eAIOK/deltatuner/setup.py
@@ -0,0 +1,53 @@
+import setuptools
+
+with open("README.md", "r") as fh:
+    long_description = fh.read()
+
+REQUIRED_PACKAGES = [
+    'torch>=1.13.0', 'transformers', 'datasets', 'sentencepiece', 
+    'peft==0.4.0', 'evaluate', 'nltk', 'rouge_score', 'einops', 
+    'sigopt', 'torchsummary'
+]
+
+setuptools.setup(
+    name="deltatuner",
+    version="0.1",
+    author="Intel AIA",
+    author_email="bdf.aiok@intel.com",
+    description="Intel extension for peft with PyTorch and DENAS",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/intel/e2eAIOK/",
+    download_url='https://github.com/intel/e2eAIOK/',
+    packages=setuptools.find_packages(
+        exclude=["example", "doc", "docker", ]),
+    python_requires=">=3.7",  # '>=3.4',  # !=3.4.*
+    install_requires=REQUIRED_PACKAGES,
+    extras_require={
+
+    },
+    entry_points={
+    },
+    classifiers=(
+        "License :: OSI Approved :: Apache Software License",
+        "Operating System :: OS Independent",
+        'Intended Audience :: Developers',
+        'Intended Audience :: Education',
+        'Intended Audience :: Science/Research',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Programming Language :: Python :: 3.10',
+        'Topic :: Scientific/Engineering',
+        'Topic :: Scientific/Engineering :: Artificial Intelligence',
+        'Topic :: Software Development',
+        'Topic :: Software Development :: Libraries',
+        'Topic :: Software Development :: Libraries :: Python Modules',
+    ),
+    license="Apache-2.0",
+    keywords=[
+        'deep learning', 'LLM', 'fine-tuning', 'pytorch', 'peft',
+        'lora', 'NAS'
+    ],
+)
\ No newline at end of file
diff --git a/tests/deltatuner/cicd/run_denas_lora.sh b/tests/deltatuner/cicd/run_denas_lora.sh
new file mode 100644
index 000000000..0d18fd22f
--- /dev/null
+++ b/tests/deltatuner/cicd/run_denas_lora.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+set -x
+mkdir -p log models
+
+# fine-tune mpt-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/mpt-7b" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/mpt_denas-lora_model \
+    --peft lora \
+    --delta lora \
+    --debugs --max_epochs 1 --population_num 1 --crossover_num 1 --mutation_num 1 --select_num 1 \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/mpt-denas-lora-run-1epoch.log
+
+# fine-tune llama2-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/Llama-2-7b-hf" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/llama2_denas-lora_model \
+    --peft lora \
+    --delta lora \
+    --debugs --max_epochs 1 --population_num 1 --crossover_num 1 --mutation_num 1 --select_num 1 \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/llama2-denas-lora-run-1epoch.log
diff --git a/tests/deltatuner/cicd/run_denas_ssf.sh b/tests/deltatuner/cicd/run_denas_ssf.sh
new file mode 100644
index 000000000..66bc9baa9
--- /dev/null
+++ b/tests/deltatuner/cicd/run_denas_ssf.sh
@@ -0,0 +1,56 @@
+set -x
+mkdir -p log
+
+DATA_PATH="/home/data"
+
+#run mpt with ssf and denas, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16" \
+        --debugs --max_epochs 1 --population_num 1 --crossover_num 1 --mutation_num 1 --select_num 1 \
+        --delta ssf \
+        --bf16 True \
+        --denas True \
+        2>&1 | tee "log/mpt-7b-ssf-allmodules-denas-bf16-run-1epoch.log"
+
+#run llama with ssf and denas, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16" \
+        --debugs --max_epochs 1 --population_num 1 --crossover_num 1 --mutation_num 1 --select_num 1 \
+        --delta ssf \
+        --denas True \
+        --bf16 True \
+        2>&1 | tee log/llama2-7b-ssf-denas-bf16-1epoch.log
\ No newline at end of file
diff --git a/tests/deltatuner/cicd/run_lora.sh b/tests/deltatuner/cicd/run_lora.sh
new file mode 100644
index 000000000..2fd00d0ff
--- /dev/null
+++ b/tests/deltatuner/cicd/run_lora.sh
@@ -0,0 +1,53 @@
+#!/bin/bash
+set -x
+mkdir -p log models
+
+# fine-tune mpt-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/mpt-7b" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/mpt_lora_model \
+    --peft lora \
+    --denas False \
+    --debugs \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/mpt-lora-run-1epoch.log
+
+# fine-tune llama2-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/Llama-2-7b-hf" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/llama2_lora_model \
+    --peft lora \
+    --denas False \
+    --debugs \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/llama2-lora-run-1epoch.log
diff --git a/tests/deltatuner/cicd/run_ssf.sh b/tests/deltatuner/cicd/run_ssf.sh
new file mode 100644
index 000000000..9661a4d8c
--- /dev/null
+++ b/tests/deltatuner/cicd/run_ssf.sh
@@ -0,0 +1,56 @@
+set -x
+mkdir -p log
+
+DATA_PATH="/home/data"
+
+#run mpt with ssf, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-bf16" \
+        --delta ssf \
+        --denas False \
+        --bf16 True \
+        --debugs \
+        2>&1 | tee log/mpt-ssf-run-allmodules-bf16-1epoch.log
+    
+#run llama2 with ssf, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-bf16" \
+        --delta ssf \
+        --denas False \
+        --bf16 True \
+        --debugs \
+        2>&1 | tee log/llama2-7b-ssf-bf16-1epoch.log
\ No newline at end of file
diff --git a/tests/deltatuner/evaluate_benchmark/cal_avg_acc.sh b/tests/deltatuner/evaluate_benchmark/cal_avg_acc.sh
new file mode 100644
index 000000000..1e9c01948
--- /dev/null
+++ b/tests/deltatuner/evaluate_benchmark/cal_avg_acc.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+set -x
+
+data=$1
+
+value=`cat $data | grep \"acc_norm\" | cut -d: -f 2 | cut -d, -f 1 | awk '{sum += $1} END {printf "NR = %d,Average = %3.4f\n",NR,sum/NR}'`
+
+echo $value
\ No newline at end of file
diff --git a/tests/deltatuner/evaluate_benchmark/eval_model.sh b/tests/deltatuner/evaluate_benchmark/eval_model.sh
new file mode 100644
index 000000000..7961770d8
--- /dev/null
+++ b/tests/deltatuner/evaluate_benchmark/eval_model.sh
@@ -0,0 +1,50 @@
+#################################################### option 1: eval llama2 on gpu ####################################################
+device=cuda:0
+pretrained_model_path=/home/vmagent/app/data/Llama-2-7b-hf
+peft_model_path=''
+output_path_prefix=/home/vmagent/app/data/llm-eval/llama2-7b-gpu
+
+bash eval_scripts.sh \
+	$device \
+	$pretrained_model_path \
+	$peft_model_path \
+	$output_path_prefix
+
+
+#################################################### option 2: eval mpt on gpu ####################################################
+device=cuda:0
+pretrained_model_path=/home/vmagent/app/data/mpt-7b
+peft_model_path=''
+output_path_prefix=/home/vmagent/app/data/llm-eval/mpt-7b-gpu
+
+bash eval_scripts.sh \
+	$device \
+	$pretrained_model_path \
+	$peft_model_path \
+	$output_path_prefix
+
+#################################################### option 3: eval llama2 on cpu ####################################################
+device=cpu
+pretrained_model_path=/home/vmagent/app/data/Llama-2-7b-hf
+peft_model_path=''
+output_path_prefix=/home/vmagent/app/data/llm-eval/llama2-7b-cpu
+
+bash eval_scripts.sh \
+	$device \
+	$pretrained_model_path \
+	$peft_model_path \
+	$output_path_prefix
+
+#################################################### option 4: eval mpt on cpu ####################################################
+device=cpu
+pretrained_model_path=/home/vmagent/app/data/mpt-7b
+peft_model_path=''
+output_path_prefix=/home/vmagent/app/data/llm-eval/mpt-7b-cpu
+
+bash eval_scripts.sh \
+	$device \
+	$pretrained_model_path \
+	$peft_model_path \
+	$output_path_prefix
+
+
diff --git a/tests/deltatuner/evaluate_benchmark/eval_scripts.sh b/tests/deltatuner/evaluate_benchmark/eval_scripts.sh
new file mode 100644
index 000000000..4916a70cc
--- /dev/null
+++ b/tests/deltatuner/evaluate_benchmark/eval_scripts.sh
@@ -0,0 +1,74 @@
+#################################################### step 1: get input args ####################################################
+# cpu, cuda:0
+device=$1
+pretrained_model_path=$2
+peft_model_path=$3
+output_path_prefix=$4
+
+
+#################################################### step 2: formulate model_args ####################################################
+# formulate model_args
+model_args='use_accelerate=True,trust_remote_code=True'
+if [ -z $peft_model_path ]; then
+    peft_model_path=""
+else
+    peft_model_path=",peft="$peft_model_path
+fi
+model_args=$model_args',pretrained='${pretrained_model_path}${peft_model_path}
+
+dtype=float16
+if [ "$device" == "cpu" ]; then
+    dtype=bfloat16
+fi
+model_args=$model_args','$dtype
+
+
+#################################################### step 3: begin evaluating ####################################################
+cd /home/vmagent/app/lm-evaluation-harness
+
+# evaluate on hellaswag
+python main.py \
+	--device $device \
+	--model hf-causal-experimental \
+	--model_args $model_args \
+	--tasks hellaswag  --num_fewshot 10 \
+	--batch_size auto --max_batch_size 32 \
+	--output_path $output_path_prefix-hellaswag
+
+# evaluate on mmlu
+python main.py \
+	--device $device \
+	--model hf-causal-experimental \
+	--model_args $model_args \
+	--tasks hendrycksTest*  --num_fewshot 5 \
+	--batch_size auto --max_batch_size 32 \
+	--output_path $output_path_prefix-mmlu
+
+# evaluate on arc_challenge
+python main.py \
+	--device $device \
+	--model hf-causal-experimental \
+	--model_args $model_args \
+	--tasks arc_challenge  --num_fewshot 25 \
+	--batch_size auto --max_batch_size 32 \
+	--output_path $output_path_prefix-arc_challenge
+
+# evaluate on truthfulqa_mc
+python main.py \
+	--device $device \
+	--model hf-causal-experimental \
+	--model_args $model_args \
+	--tasks truthfulqa_mc  --num_fewshot 0 \
+	--batch_size auto --max_batch_size 32 \
+	--output_path $output_path_prefix-truthqa
+
+
+#################################################### step 4: calculate average acc of MMLU task ####################################################
+
+cd /home/vmagent/app/deltatuner/tests/evaluate_benchmark
+bash cal_avg_acc.sh $output_path_prefix-mmlu
+
+
+
+
+
diff --git a/tests/deltatuner/finetune/run_denas_lora.sh b/tests/deltatuner/finetune/run_denas_lora.sh
new file mode 100644
index 000000000..801f4e6da
--- /dev/null
+++ b/tests/deltatuner/finetune/run_denas_lora.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+set -x
+mkdir -p log models
+
+# fine-tune mpt-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/mpt-7b" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/mpt_denas-lora_model \
+    --peft lora \
+    --delta lora \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/mpt-denas-lora-run-1epoch.log
+
+# fine-tune llama2-7b with denas-lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/Llama-2-7b-hf" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/llama2_denas-lora_model \
+    --peft lora \
+    --delta lora \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/llama2-denas-lora-run-1epoch.log
+
+
+# used for assign target modules to lora
+# --lora_target_modules q_proj v_proj k_proj o_proj up_proj down_proj \
+# --lora_target_modules Wqkv out_proj up_proj down_proj \
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/run_lora.sh b/tests/deltatuner/finetune/run_lora.sh
new file mode 100644
index 000000000..ba4b331ae
--- /dev/null
+++ b/tests/deltatuner/finetune/run_lora.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+set -x
+mkdir -p log models
+
+# fine-tune mpt-7b with lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/mpt-7b" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/mpt_lora_model \
+    --peft lora \
+    --denas False \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/mpt-lora-run-1epoch.log
+
+# fine-tune llama2-7b with lora
+python example/instruction_tuning_pipeline/finetune_clm.py \
+    --model_name_or_path "/home/vmagent/app/dataset/Llama-2-7b-hf" \
+    --train_file "/home/vmagent/app/dataset/stanford_alpaca/alpaca_data.json" \
+    --dataset_concatenation \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 8 \
+    --gradient_accumulation_steps 1 \
+    --do_train \
+    --do_eval \
+    --validation_split_percentage 30 \
+    --learning_rate 1e-4 \
+    --num_train_epochs 1 \
+    --logging_steps 100 \
+    --save_total_limit 1 \
+    --log_level info \
+    --save_strategy epoch \
+    --output_dir models/llama2_lora_model \
+    --peft lora \
+    --denas False \
+    --trust_remote_code True \
+    --no_cuda \
+    --bf16 True 2>&1 | tee log/llama2-lora-run-1epoch.log
+
+
+# used for assign target modules to lora
+# --lora_target_modules q_proj v_proj k_proj o_proj up_proj down_proj \
+# --lora_target_modules Wqkv out_proj up_proj down_proj \
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/ssf/llama2-7b-lora.sh b/tests/deltatuner/finetune/ssf/llama2-7b-lora.sh
new file mode 100755
index 000000000..662074530
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/llama2-7b-lora.sh
@@ -0,0 +1,24 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-lora" \
+        --peft lora \
+        --denas False \
+        | tee llama2-7b-lora-1epoch.log
diff --git a/tests/deltatuner/finetune/ssf/llama2-7b-ssf-allmodules.sh b/tests/deltatuner/finetune/ssf/llama2-7b-ssf-allmodules.sh
new file mode 100755
index 000000000..7077fda90
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/llama2-7b-ssf-allmodules.sh
@@ -0,0 +1,135 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-allmodules" \
+        --ssf_target_modules q_proj v_proj k_proj o_proj up_proj down_proj \
+        --delta ssf \
+        --denas False \
+        | tee llama2-7b-ssf-allmodules-1epoch.log
+
+#bf16  
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-allmodules-bf16" \
+        --ssf_target_modules q_proj v_proj k_proj o_proj up_proj down_proj \
+        --delta ssf \
+        --bf16 True \
+        --denas False \
+        | tee llama2-7b-ssf-allmodules-bf16-1epoch.log
+
+#merge
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-allmodules-merge" \
+        --ssf_target_modules q_proj v_proj k_proj o_proj up_proj down_proj \
+        --delta ssf \
+        --resume_peft "$DATA_PATH/llama2-7b-ssf-allmodules" \
+        --save_merged_model True
+
+#merge, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-allmodules-bf16-merge" \
+        --ssf_target_modules q_proj v_proj k_proj o_proj up_proj down_proj \
+        --delta ssf \
+        --resume_peft "$DATA_PATH/llama2-7b-ssf-allmodules-bf16" \
+        --save_merged_model True \
+        --bf16 True 
+
+#evaluate merged model
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/llama2-7b-ssf-allmodules-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-allmodules-merge/eval_merged" \
+        --debugs
+
+#evaluate merged model, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/llama2-7b-ssf-allmodules-bf16-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-allmodules-bf16-merge/eval_merged" \        
+        --bf16 True \
+        --debugs
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/ssf/llama2-7b-ssf-denas.sh b/tests/deltatuner/finetune/ssf/llama2-7b-ssf-denas.sh
new file mode 100755
index 000000000..aa6fb3704
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/llama2-7b-ssf-denas.sh
@@ -0,0 +1,157 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas" \
+        --delta ssf \
+        --denas True \
+        | tee llama2-7b-ssf-denas-1epoch.log
+
+#bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16" \
+        --delta ssf \
+        --denas True \
+        --bf16 True \
+        | tee llama2-7b-ssf-denas-bf16-1epoch.log
+
+#evaluate model, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16/eval" \
+        --delta ssf \
+        --bf16 True \
+        --resume_peft "$DATA_PATH/llama2-7b-ssf-denas-bf16" \
+        --denas "$DATA_PATH/llama2-7b-ssf-denas-bf16/best_model_structure.txt" \
+        --debugs
+
+#merge
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-merge" \
+        --delta ssf \
+        --resume_peft "$DATA_PATH/llama2-7b-ssf-denas" \
+        --save_merged_model True \
+        --denas "$DATA_PATH/llama2-7b-ssf-denas/best_model_structure.txt" \
+        --debugs
+
+#merge, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16-merge" \
+        --delta ssf \
+        --bf16 True \
+        --resume_peft "$DATA_PATH/llama2-7b-ssf-denas-bf16" \
+        --save_merged_model True \
+        --denas "$DATA_PATH/llama2-7b-ssf-denas-bf16/best_model_structure.txt" \
+        --debugs
+
+#evaluate merged model
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/llama2-7b-ssf-denas-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-merge/eval_merge" \
+        --debugs
+
+#evaluate merged model, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/llama2-7b-ssf-denas-bf16-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16-merge/eval_merge" \
+        --bf16 True \
+        --debugs
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/ssf/llama2-7b-ssf.sh b/tests/deltatuner/finetune/ssf/llama2-7b-ssf.sh
new file mode 100755
index 000000000..39299cf36
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/llama2-7b-ssf.sh
@@ -0,0 +1,49 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf" \
+        --delta ssf \
+        --denas False \
+        | tee llama2-7b-ssf-1epoch.log
+
+#bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-bf16" \
+        --delta ssf \
+        --bf16 True \
+        --denas False \
+        | tee llama2-7b-ssf-bf16-1epoch.log
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/ssf/mpt-7b-full.sh b/tests/deltatuner/finetune/ssf/mpt-7b-full.sh
new file mode 100755
index 000000000..27211fc0d
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/mpt-7b-full.sh
@@ -0,0 +1,23 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-full" \
+        | tee mpt-lora-run-1epoch.log
diff --git a/tests/deltatuner/finetune/ssf/mpt-7b-lora.sh b/tests/deltatuner/finetune/ssf/mpt-7b-lora.sh
new file mode 100755
index 000000000..8b9e695f3
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/mpt-7b-lora.sh
@@ -0,0 +1,24 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-lora" \
+        --peft lora \
+        | tee mpt-lora-run-1epoch.log
diff --git a/tests/deltatuner/finetune/ssf/mpt-7b-ssf-allmodules-denas.sh b/tests/deltatuner/finetune/ssf/mpt-7b-ssf-allmodules-denas.sh
new file mode 100755
index 000000000..251152050
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/mpt-7b-ssf-allmodules-denas.sh
@@ -0,0 +1,120 @@
+DATA_PATH="/home/data"
+
+#bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16" \
+        --delta ssf \
+        --ssf_target_modules Wqkv out_proj up_proj down_proj \
+        --bf16 True \
+        --denas True \
+        | tee "mpt-7b-ssf-allmodules-denas-bf16-run-1epoch.log"
+
+#merge
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16-merge" \
+        --delta ssf \
+        --ssf_target_modules Wqkv out_proj up_proj down_proj \
+        --resume_peft "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16" \
+        --save_merged_model True \
+        --denas "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16/best_model_structure.txt" \
+        --debugs
+
+#merge,bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16-merge" \
+        --delta ssf \
+        --ssf_target_modules Wqkv out_proj up_proj down_proj \
+        --bf16 True \
+        --resume_peft "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16" \
+        --save_merged_model True \
+        --denas "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16/best_model_structure.txt" \
+        --debugs
+
+#evaluate merged model
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16-merge/eval_merge" \
+        --debugs
+
+#evaluate merged model, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-denas-bf16-merge/eval_merge" \
+        --bf16 True \
+        --debugs
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/ssf/mpt-7b-ssf-allmodules.sh b/tests/deltatuner/finetune/ssf/mpt-7b-ssf-allmodules.sh
new file mode 100755
index 000000000..ff9f34385
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/mpt-7b-ssf-allmodules.sh
@@ -0,0 +1,98 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules" \
+        --delta ssf \
+        --denas False \
+        --ssf_target_modules Wqkv out_proj up_proj down_proj \
+        | tee mpt-ssf-run-allmodules-1epoch.log
+
+# bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-bf16" \
+        --delta ssf \
+        --denas False \
+        --ssf_target_modules Wqkv out_proj up_proj down_proj \
+        --bf16 True \
+        | tee mpt-ssf-run-allmodules-bf16-1epoch.log
+
+# merge, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-bf16-merge" \
+        --delta ssf \
+        --ssf_target_modules Wqkv out_proj up_proj down_proj \
+        --bf16 True \
+        --resume_peft "$DATA_PATH/mpt-7b-ssf-allmodules-bf16" \
+        --save_merged_model True
+
+#evaluate merged model, bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b-ssf-allmodules-bf16-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-allmodules-bf16-merge/eval_merged" \
+        --bf16 True \
+        --debugs
diff --git a/tests/deltatuner/finetune/ssf/mpt-7b-ssf.sh b/tests/deltatuner/finetune/ssf/mpt-7b-ssf.sh
new file mode 100755
index 000000000..f4d301b45
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/mpt-7b-ssf.sh
@@ -0,0 +1,51 @@
+DATA_PATH="/home/data"
+
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf" \
+        --denas False \
+        --delta ssf \
+        | tee mpt-ssf-run-1epoch.log
+
+# bf16
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/mpt-7b" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --tokenizer_name "$DATA_PATH/gpt-neox-20b" \
+        --no_cuda \
+        --output_dir "$DATA_PATH/mpt-7b-ssf-bf16" \
+        --delta ssf \
+        --bf16 True \
+        --denas False \
+        | tee mpt-ssf-run-1epoch.log
\ No newline at end of file
diff --git a/tests/deltatuner/finetune/ssf/readme.md b/tests/deltatuner/finetune/ssf/readme.md
new file mode 100644
index 000000000..25bcbd9c4
--- /dev/null
+++ b/tests/deltatuner/finetune/ssf/readme.md
@@ -0,0 +1,98 @@
+# Run Deltatuner with ssf
+
+## Llama2-7b
+
+### 1. Finetune the model
+If use Denas, can use command:
+
+```shell
+cd frameworks.bigdata.AIDK/example
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_train \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16" \
+        --delta ssf \
+        --denas True \
+        --bf16 True \
+        | tee llama2-7b-ssf-denas-bf16-1epoch.log
+```
+
+### 2. Save merged model
+Merge the SSF adapter into orginal base model.
+
+```shell
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/Llama-2-7b-hf" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_train_batch_size 8 \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --learning_rate 1e-4 \
+        --num_train_epochs 1 \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --save_strategy epoch \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16-merge" \
+        --delta ssf \
+        --bf16 True \
+        --resume_peft "$DATA_PATH/llama2-7b-ssf-denas-bf16" \
+        --save_merged_model True \
+        --denas "$DATA_PATH/llama2-7b-ssf-denas-bf16/best_model_structure.txt" \
+        --debugs
+```
+
+### 3. Evaluate merged model
+As ssf will enable bias, while the default Llama2 disable all the bias, to enable the full parameters of the adapter, we need to change the model definition.
+
+First copy the updated model codes with the merged weights.
+```shell
+cp instruction_tuning_pipeline/models/llama2-ssf/* $DATA_PATH/llama2-7b-ssf-denas-bf16-merge/merged_model
+```
+
+Then update the "best_model_structure" and "target_modules" setting in config.json. if not enable denas and not change "target_modules" default settings, can skip correpsonding setting.
+```shell
+...
+"best_model_structure":  {"num_hidden_layers": [1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1]}, # change to your best structure, skip to keep default
+"target_modules": ["q_proj", "v_proj"], #change to your setting, skip to keep default
+...
+```
+
+Finally we can directly evalute the merged model.
+```shell
+python instruction_tuning_pipeline/finetune_clm.py \
+        --model_name_or_path "$DATA_PATH/llama2-7b-ssf-denas-bf16-merge/merged_model" \
+        --train_file "$DATA_PATH/alpaca_data.json" \
+        --dataset_concatenation \
+        --per_device_eval_batch_size 8 \
+        --gradient_accumulation_steps 1 \
+        --validation_split_percentage 30 \
+        --do_eval \
+        --logging_steps 100 \
+        --save_total_limit 1 \
+        --log_level info \
+        --trust_remote_code True \
+        --no_cuda \
+        --output_dir "$DATA_PATH/llama2-7b-ssf-denas-bf16-merge/eval_merge" \
+        --bf16 True
+```
\ No newline at end of file
diff --git a/tests/unittest/test_denas_asr_model_builder.py b/tests/unittest/denas/test_denas_asr_model_builder.py
similarity index 100%
rename from tests/unittest/test_denas_asr_model_builder.py
rename to tests/unittest/denas/test_denas_asr_model_builder.py
diff --git a/tests/unittest/test_denas_cv_trainer.py b/tests/unittest/denas/test_denas_cv_trainer.py
similarity index 100%
rename from tests/unittest/test_denas_cv_trainer.py
rename to tests/unittest/denas/test_denas_cv_trainer.py
diff --git a/tests/unittest/test_denas_evolutionary.py b/tests/unittest/denas/test_denas_evolutionary.py
similarity index 100%
rename from tests/unittest/test_denas_evolutionary.py
rename to tests/unittest/denas/test_denas_evolutionary.py
diff --git a/tests/unittest/test_denas_random.py b/tests/unittest/denas/test_denas_random.py
similarity index 100%
rename from tests/unittest/test_denas_random.py
rename to tests/unittest/denas/test_denas_random.py
diff --git a/tests/unittest/test_ma_adapter.py b/tests/unittest/ma/test_ma_adapter.py
similarity index 100%
rename from tests/unittest/test_ma_adapter.py
rename to tests/unittest/ma/test_ma_adapter.py
diff --git a/tests/unittest/test_ma_distiller.py b/tests/unittest/ma/test_ma_distiller.py
similarity index 100%
rename from tests/unittest/test_ma_distiller.py
rename to tests/unittest/ma/test_ma_distiller.py
diff --git a/tests/unittest/test_ma_finetuner.py b/tests/unittest/ma/test_ma_finetuner.py
similarity index 100%
rename from tests/unittest/test_ma_finetuner.py
rename to tests/unittest/ma/test_ma_finetuner.py
diff --git a/tests/unittest/test_ma_transferrable_model.py b/tests/unittest/ma/test_ma_transferrable_model.py
similarity index 100%
rename from tests/unittest/test_ma_transferrable_model.py
rename to tests/unittest/ma/test_ma_transferrable_model.py
diff --git a/tests/unittest/init_hydro_for_test.py b/tests/unittest/utils/init_hydro_for_test.py
similarity index 100%
rename from tests/unittest/init_hydro_for_test.py
rename to tests/unittest/utils/init_hydro_for_test.py
diff --git a/tests/unittest/test_hydrodb.py b/tests/unittest/utils/test_hydrodb.py
similarity index 100%
rename from tests/unittest/test_hydrodb.py
rename to tests/unittest/utils/test_hydrodb.py