Repository: RUCAIBox/RecBole (language: python)

Issue:
FIX & FEA: Add SINE in sequential and fix random state in hypertuning
- Add SINE model in sequential models based on #847 
- Update number of models to 77 (including SINE, NCE-PLRec and ADMMSLIM)
- Fix `AttributeError` in hypertuning: `np.random.RandomState` was deprecated, use `np.random.default_rng` instead


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index 0648ec38d..153d3be8b 100644
--- a/README.md
+++ b/README.md
@@ -23,7 +23,7 @@
 
 RecBole is developed based on Python and PyTorch for reproducing and developing recommendation algorithms in a unified,
 comprehensive and efficient framework for research purpose.
-Our library includes 72 recommendation algorithms, covering four major categories:
+Our library includes 77 recommendation algorithms, covering four major categories:
 
 + General Recommendation
 + Sequential Recommendation
@@ -46,7 +46,7 @@ by our team.
 + **General and extensible data structure.** We design general and extensible data structures to unify the formatting and
 usage of various recommendation datasets.
 
-+ **Comprehensive benchmark models and datasets.** We implement 72 commonly used recommendation algorithms, and provide
++ **Comprehensive benchmark models and datasets.** We implement 77 commonly used recommendation algorithms, and provide
 the formatted copies of 28 recommendation datasets.
 
 + **Efficient GPU-accelerated execution.** We optimize the efficiency of our library with a number of improved techniques
diff --git a/README_CN.md b/README_CN.md
index 84bbec434..67bc60ba1 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -23,7 +23,7 @@
 
 
 RecBole 是一个基于 PyTorch 实现的，面向研究者的，易于开发与复现的，统一、全面、高效的推荐系统代码库。
-我们实现了72个推荐系统模型，包含常见的推荐系统类别，如:
+我们实现了77个推荐系统模型，包含常见的推荐系统类别，如:
 
 + General Recommendation
 + Sequential Recommendation
@@ -45,7 +45,7 @@ RecBole 是一个基于 PyTorch 实现的，面向研究者的，易于开发与
 ## 特色
 + **通用和可扩展的数据结构** 我们设计了通用和可扩展的数据结构来支持各种推荐数据集统一化格式和使用。
 
-+ **全面的基准模型和数据集** 我们实现了72个常用的推荐算法，并提供了28个推荐数据集的格式化副本。
++ **全面的基准模型和数据集** 我们实现了77个常用的推荐算法，并提供了28个推荐数据集的格式化副本。
 
 + **高效的 GPU 加速实现** 我们针对GPU环境使用了一系列的优化技术来提升代码库的效率。
 
diff --git a/docs/source/asset/lightsans.png b/docs/source/asset/lightsans.png
new file mode 100644
index 000000000..86a878385
Binary files /dev/null and b/docs/source/asset/lightsans.png differ
diff --git a/docs/source/asset/sgl.png b/docs/source/asset/sgl.png
new file mode 100644
index 000000000..db95d7adc
Binary files /dev/null and b/docs/source/asset/sgl.png differ
diff --git a/docs/source/asset/sine.png b/docs/source/asset/sine.png
new file mode 100644
index 000000000..3fa12d0a5
Binary files /dev/null and b/docs/source/asset/sine.png differ
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 40cd02700..9dc0d746a 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -11,7 +11,7 @@ Introduction
 RecBole is a unified, comprehensive and efficient framework developed based on PyTorch.
 It aims to help the researchers to reproduce and develop recommendation models.
 
-In the lastest release, our library includes 73 recommendation algorithms `[Model List]`_, covering four major categories:
+In the lastest release, our library includes 77 recommendation algorithms `[Model List]`_, covering four major categories:
 
 - General Recommendation
 - Sequential Recommendation
@@ -29,7 +29,7 @@ Features:
 - General and extensible data structure
     We deign general and extensible data structures to unify the formatting and usage of various recommendation datasets.
 - Comprehensive benchmark models and datasets
-    We implement 73 commonly used recommendation algorithms, and provide the formatted copies of 28 recommendation datasets.
+    We implement 77 commonly used recommendation algorithms, and provide the formatted copies of 28 recommendation datasets.
 - Efficient GPU-accelerated execution
     We design many tailored strategies in the GPU environment to enhance the efficiency of our library.
 - Extensive and standard evaluation protocols
diff --git a/docs/source/recbole/recbole.model.general_recommender.rst b/docs/source/recbole/recbole.model.general_recommender.rst
index b4cc32610..4b5a73231 100644
--- a/docs/source/recbole/recbole.model.general_recommender.rst
+++ b/docs/source/recbole/recbole.model.general_recommender.rst
@@ -22,3 +22,4 @@ recbole.model.general\_recommender
    recbole.model.general_recommender.ngcf
    recbole.model.general_recommender.pop
    recbole.model.general_recommender.spectralcf
+   recbole.model.general_recommender.sgl
\ No newline at end of file
diff --git a/docs/source/recbole/recbole.model.general_recommender.sgl.rst b/docs/source/recbole/recbole.model.general_recommender.sgl.rst
new file mode 100644
index 000000000..4b3622941
--- /dev/null
+++ b/docs/source/recbole/recbole.model.general_recommender.sgl.rst
@@ -0,0 +1,4 @@
+.. automodule:: recbole.model.general_recommender.sgl
+   :members:
+   :undoc-members:
+   :show-inheritance:
\ No newline at end of file
diff --git a/docs/source/user_guide/config/environment_settings.rst b/docs/source/user_guide/config/environment_settings.rst
index 15fb52bee..19b8edf55 100644
--- a/docs/source/user_guide/config/environment_settings.rst
+++ b/docs/source/user_guide/config/environment_settings.rst
@@ -23,9 +23,17 @@ Environment settings are designed to set basic parameters of running environment
 - ``save_dataset (bool)``: Whether or not save filtered dataset.
   If True, save filtered dataset, otherwise it will not be saved.
   Defaults to ``False``.
+- ``dataset_save_path (str)``: The path of saved dataset. The tool will attempt to load the dataset from this path.
+  If it equals to ``None``, the tool will try to load the dataset from ``{checkpoint_dir}/{dataset}-{dataset_class_name}.pth``.
+  If the config of saved dataset is not equal to current config, the tool will create dataset from scratch.
+  Defaults to ``None``.
 - ``save_dataloaders (bool)``: Whether or not save split dataloaders.
   If True, save split dataloaders, otherwise they will not be saved.
   Defaults to ``False``.
+- ``dataloaders_save_path (str)``: The path of saved dataloaders. The tool will attempt to load the dataloaders from this path.
+  If it equals to ``None``, the tool will try to load the dataloaders from ``{checkpoint_dir}/{dataset}-for-{model}-dataloader.pth``.
+  If the config of saved dataloaders is not equal to current config, the tool will create dataloaders from scratch.
+  Defaults to ``None``.
 - ``log_wandb (bool)``: Whether or not use Weights & Biases(W&B).
   If True, use W&B to visualize configs and metrics of different experiments, otherwise it will not be used.
   Defaults to ``False``.
diff --git a/docs/source/user_guide/config/training_settings.rst b/docs/source/user_guide/config/training_settings.rst
index f009c861c..1d4ff583f 100644
--- a/docs/source/user_guide/config/training_settings.rst
+++ b/docs/source/user_guide/config/training_settings.rst
@@ -9,10 +9,15 @@ Training settings are designed to set parameters about model training.
   Range in ``['adam', 'sgd', 'adagrad', 'rmsprop', 'sparse_adam']``.
 - ``learning_rate (float)`` : Learning rate. Defaults to ``0.001``.
 - ``neg_sampling(dict)``: This parameter controls the negative sampling for model training.
-  The key range is ``['uniform', 'popularity']``, which decides the distribution of negative items in sampling pools.
+  The key range is ``['uniform', 'popularity']``, which decides the distribution of negative items in sampling pools. In addition, we also support dynamic negative sampling ``['dynamic']``.
   ``uniform`` means uniformly select negative items while ``popularity`` means select negative items based on 
-  their popularity (Counter(item) in `.inter` file). The value k (int) of this parameter means sample k negative items for each positive item. Note that if your data is labeled, you need to set this parameter as ``None``.
-  The default value of this parameter is ``{'uniform': 1}``, which means uniformly sample one negative item for each positive item. 
+  their popularity (Counter(item) in `.inter` file). The value k (int) of this parameter means sample k negative items for each positive item.
+  Note that if your data is labeled, you need to set this parameter as ``None``.
+  The default value of this parameter is ``{'uniform': 1}``, which means uniformly sample one negative item for each positive item.
+  When dynamic negative sampling, ``dynamic`` decides the number of candidate negative items.
+  For example, ``{'uniform': 1, 'dynamic': 2}`` means sampling 2 items for each positive item uniformly,
+  and dynamically choosing the item with the higher score as the selected negative item.
+  In particular, ``'uniform': 1`` means that a positive item pairs with one negative item, and ``'dynamic': 2`` means dynamically selecting each negative item from two candidates.
 - ``eval_step (int)`` : The number of training epochs before an evaluation
   on the valid dataset. If it is less than 1, the model will not be
   evaluated on the valid dataset. Defaults to ``1``.
diff --git a/docs/source/user_guide/model/general/admmslim.rst b/docs/source/user_guide/model/general/admmslim.rst
new file mode 100644
index 000000000..d419adddd
--- /dev/null
+++ b/docs/source/user_guide/model/general/admmslim.rst
@@ -0,0 +1,102 @@
+ADMMSLIM
+============
+
+Introduction
+------------------
+
+`[paper] <https://doi.org/10.1145/3336191.3371774>`_
+
+**Title:** ADMM SLIM: Sparse Recommendations for Many Users
+
+**Authors:** Harald Steck,Maria Dimakopoulou,Nickolai Riabov,Tony Jebara
+
+
+**Abstract:** The Sparse Linear Method (Slim) is a well-established approach
+for top-N recommendations. This article proposes several improvements
+that are enabled by the Alternating Directions Method of
+Multipliers (ADMM), a well-known optimization method
+with many application areas. First, we show that optimizing the
+original Slim-objective by ADMM results in an approach where the
+training time is independent of the number of users in the training
+data, and hence trivially scales to large numbers of users. Second,
+the flexibility of ADMM allows us to switch on and off the various
+constraints and regularization terms in the original Slim-objective,
+in order to empirically assess their contributions to ranking accuracy
+on given data. Third, we also propose two extensions to the
+original Slim training-objective in order to improve recommendation
+accuracy further without increasing the computational cost. In
+our experiments on three well-known data-sets, we first compare
+to the original Slim-implementation and find that not only ADMM
+reduces training time considerably, but also achieves an improvement
+in recommendation accuracy due to better optimization. We
+then compare to various state-of-the-art approaches and observe
+up to 25% improvement in recommendation accuracy in our experiments.
+Finally, we evaluate the importance of sparsity and the
+non-negativity constraint in the original Slim-objective with subsampling
+experiments that simulate scenarios of cold-starting and
+large catalog sizes compared to relatively small user base, which
+often occur in practice.
+
+Running with RecBole
+-------------------------
+
+**Model Hyper-Parameters:**
+
+- ``lambda1 (float)`` : L1-norm regularization parameter. Defaults to ``3``.
+
+- ``lambda2 (float)`` : L2-norm regularization parameter. Defaults to ``200``.
+
+- ``alpha (float)`` : The exponents to control the power-law in the regularization terms. Defaults to ``0.5``.
+
+- ``rho (float)`` : The penalty parameter that applies to the squared difference between primal variables. Defaults to ``4000``.
+
+- ``k (int)`` : The number of running iterations. Defaults to ``100``.
+
+- ``positive_only (bool)`` : Whether only preserves all positive values. Defaults to ``True``.
+
+- ``center_columns (bool)`` : Whether to use additional item-bias terms.. Defaults to ``False``.
+
+
+**A Running Example:**
+
+Write the following code to a python file, such as `run.py`
+
+.. code:: python
+
+   from recbole.quick_start import run_recbole
+
+   run_recbole(model='ADMMSLIM', dataset='ml-100k')
+
+And then:
+
+.. code:: bash
+
+   python run.py
+
+Tuning Hyper Parameters
+-------------------------
+
+If you want to use ``HyperTuning`` to tune hyper parameters of this model, you can copy the following settings and name it as ``hyper.test``.
+
+.. code:: bash
+
+    lambda1 choice [0.1 , 0.5 , 5 , 10]
+    lambda2 choice [5 , 50 , 1000 , 5000]
+    alpha choice [0.25 , 0.5 , 0.75 , 1]
+
+Note that we just provide these hyper parameter ranges for reference only, and we can not guarantee that they are the optimal range of this model.
+
+Then, with the source code of RecBole (you can download it from GitHub), you can run the ``run_hyper.py`` to tuning:
+
+.. code:: bash
+
+	python run_hyper.py --model=[model_name] --dataset=[dataset_name] --config_files=[config_files_path] --params_file=hyper.test
+
+For more details about Parameter Tuning, refer to :doc:`../../../user_guide/usage/parameter_tuning`.
+
+If you want to change parameters, dataset or evaluation settings, take a look at
+
+- :doc:`../../../user_guide/config_settings`
+- :doc:`../../../user_guide/data_intro`
+- :doc:`../../../user_guide/train_eval_intro`
+- :doc:`../../../user_guide/usage`
\ No newline at end of file
diff --git a/docs/source/user_guide/model/general/sgl.rst b/docs/source/user_guide/model/general/sgl.rst
new file mode 100644
index 000000000..1544fc6b4
--- /dev/null
+++ b/docs/source/user_guide/model/general/sgl.rst
@@ -0,0 +1,106 @@
+SGL
+============
+
+Introduction
+------------------
+
+`[paper] <https://dl.acm.org/doi/10.1145/3404835.3462862>`_
+
+**Title:** Self-supervised Graph Learning for Recommendation
+
+**Authors:** Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie
+
+
+**Abstract:**
+Representation learning on user-item graph for recommendation
+has evolved from using single ID or interaction history to
+exploiting higher-order neighbors. This leads to the success of graph
+convolution networks (GCNs) for recommendation such as PinSage
+and LightGCN. Despite effectiveness, we argue that they suffer
+from two limitations: (1) high-degree nodes exert larger impact on
+the representation learning, deteriorating the recommendations of
+low-degree (long-tail) items; and (2) representations are vulnerable
+to noisy interactions, as the neighborhood aggregation scheme
+further enlarges the impact of observed edges.
+In this work, we explore self-supervised learning on useritem
+graph, so as to improve the accuracy and robustness
+of GCNs for recommendation. The idea is to supplement the
+classical supervised task of recommendation with an auxiliary selfsupervised
+task, which reinforces node representation learning
+via self-discrimination. Specifically, we generate multiple views
+of a node, maximizing the agreement between different views of
+the same node compared to that of other nodes. We devise three
+operators to generate the views — node dropout, edge dropout,
+and random walk — that change the graph structure in different
+manners. We term this new learning paradigm as Self-supervised
+Graph Learning (SGL), implementing it on the state-of-the-art model
+LightGCN. Through theoretical analyses, we find that SGL has the
+ability of automatically mining hard negatives. Empirical studies
+on three benchmark datasets demonstrate the effectiveness of
+SGL, which improves the recommendation accuracy, especially on
+long-tail items, and the robustness against interaction noises.
+
+
+.. image:: ../../../asset/sgl.png
+    :width: 500
+    :align: center
+
+Running with RecBole
+-------------------------
+
+**Model Hyper-Parameters:**
+
+- ``type (str)`` : The operators to generate the views. ``"ED"`` stands for edge dropout mode. ``"ND"`` stands for node dropout mode. ``"RW"`` stands for random walk mode. Defaults to ``"ED"``.
+- ``n_layers (int)`` : The number of layers in SGL. Defaults to ``3``.
+- ``ssl_tau (float)`` : The temperature in softmax. Defaults to ``0.5``.
+- ``embedding_size (int)`` : the embedding size of users and items. Defaults to ``64``.
+- ``drop_ratio (float)`` The dropout ratio. Defaults to ``0.1``.
+- ``reg_weight (float)`` : The L2 regularization weight. Defaults to ``1e-05``.
+- ``ssl_weight (float)`` : The hyperparameters to control the strengths of SSL. Defaults to ``0.05``.
+
+**A Running Example:**
+
+Write the following code to a python file, such as `run.py`
+
+.. code:: python
+
+   from recbole.quick_start import run_recbole
+
+   run_recbole(model='SGL', dataset='ml-100k')
+
+And then:
+
+.. code:: bash
+
+   python run.py
+
+
+Tuning Hyper Parameters
+-------------------------
+
+If you want to use ``HyperTuning`` to tune hyper parameters of this model, you can copy the following settings and name it as ``hyper.test``.
+
+.. code:: bash
+
+   ssl_tau choice [0.1,0.2,0.5,1.0]
+   drop_ratio choice [0,0.1,0.2,0.4,0.5]
+   ssl_weight choice [0.005,0.05,0.1,0.5,1.0]
+
+
+Note that we just provide these hyper parameter ranges for reference only, and we can not guarantee that they are the optimal range of this model.
+
+Then, with the source code of RecBole (you can download it from GitHub), you can run the ``run_hyper.py`` to tuning:
+
+.. code:: bash
+
+	python run_hyper.py --model=[model_name] --dataset=[dataset_name] --config_files=[config_files_path] --params_file=hyper.test
+
+For more details about Parameter Tuning, refer to :doc:`../../../user_guide/usage/parameter_tuning`.
+
+
+If you want to change parameters, dataset or evaluation settings, take a look at
+
+- :doc:`../../../user_guide/config_settings`
+- :doc:`../../../user_guide/data_intro`
+- :doc:`../../../user_guide/train_eval_intro`
+- :doc:`../../../user_guide/usage`
\ No newline at end of file
diff --git a/docs/source/user_guide/model/sequential/lightsans.rst b/docs/source/user_guide/model/sequential/lightsans.rst
new file mode 100644
index 000000000..56ac75586
--- /dev/null
+++ b/docs/source/user_guide/model/sequential/lightsans.rst
@@ -0,0 +1,95 @@
+LightSANs
+=================
+
+Introduction
+---------------------
+
+`[paper] <https://doi.org/10.1145/3404835.3462978>`_
+
+**Title:** Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation
+
+**Authors:** Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Xin Zhao, Xing Xie, and Ji-Rong Wen
+
+**Abstract:**  Self-attention networks (SANs) have been intensively applied for sequential recommenders,
+but they are limited due to: (1) the quadratic complexity and vulnerability to over-parameterization
+in self-attention; (2) inaccurate modeling of sequential relations between items due
+to the implicit position encoding. In this work, we propose the low-rank decomposed
+self-attention networks (LightSANs) to overcome these problems. Particularly, we introduce
+the low-rank decomposed self-attention, which projects user's historical items into
+a small constant number of latent interests and leverages item-to-interest interaction
+to generate the context-aware representation. It scales linearly w.r.t. the user's
+historical sequence length in terms of time and space, and is more resilient to over-parameterization.
+Besides, we design the decoupled position encoding, which models the sequential relations
+between items more precisely. Extensive experimental studies are carried out on three
+real-world datasets, where LightSANs outperform the existing SANs-based recommenders
+in terms of both effectiveness and efficiency.
+
+.. image:: ../../../asset/lightsans.png
+    :width: 500
+    :align: center
+
+Running with RecBole
+-------------------------
+
+**Model Hyper-Parameters:**
+
+- ``num_layers (int)`` : The number of transformer layers in transformer encoder. Default to ``2``.
+- ``n_heads (int)`` : The number of attention heads for multi-head attention layer. Defaults to ``2``.
+- ``k_interests (int)`` : The number of latent interests dimension in item-to-interest aggregation. Defaults to ``5``. We recommend setting `k_interests=0.1*MAX_ITEM_LIST_LENGTH` to ensure the compression effect.
+- ``hidden_size (int)`` : The input and output hidden size. Defaults to ``64``.
+- ``inner_size (int)`` : The imensionality in feed-forward layer. Defaults to ``256``.
+- ``hidden_dropout_prob (float)``: The probability of an element to be zeroed. Defaults to ``0.5``.
+- ``attn_dropout_prob (float)``: The probability of an attention score to be zeroed. Defaults to ``0.5``.
+- ``hidden_act (str)`` : The activation function in feed-forward layer. Defaults to ``'gelu'``. Range in ``['gelu', 'relu', 'swish', 'tanh', 'sigmoid']``.
+- ``layer_norm_eps (float)``: The value added to the denominator for numerical stability. Defaults to ``1e-12``.
+- ``initializer_range (float)``: The range of weights initialization. Defaults to ``0.02``.
+- ``loss_type (str)`` : The type of loss function. If ``loss_type='CE'``, the training task is regarded as a multi-classification task and the target item is the ground truth. In this way, negative sampling is not needed. If ``loss_type='BPR'``, the training task will be optimized in the pair-wise way, which maximizes the difference between positive item and negative item. In this way, negative sampling is necessary, such as setting ``--neg_sampling="{'uniform': 1}"``. Defaults to ``'CE'``. Range in ``['BPR', 'CE']``.
+
+
+**A Running Example:**
+
+Write the following code to a python file, such as `run.py`
+
+.. code:: python
+
+   from recbole.quick_start import run_recbole
+
+   parameter_dict = {
+      'neg_sampling': None,
+   }
+   run_recbole(model='LightSANs', dataset='ml-100k', config_dict=parameter_dict)
+
+And then:
+
+.. code:: bash
+
+   python run.py
+
+Tuning Hyper Parameters
+-------------------------
+
+If you want to use ``HyperTuning`` to tune hyper parameters of this model, you can copy the following settings and name it as ``hyper.test``.
+
+.. code:: bash
+
+   learning_rate choice [0.01,0.005,0.001,0.0005,0.0001]
+   num_layers choice [1,2,3]
+   k_interests choice [10,15,20]
+
+Note that we just provide these hyper parameter ranges for reference only, and we can not guarantee that they are the optimal range of this model.
+
+Then, with the source code of RecBole (you can download it from GitHub), you can run the ``run_hyper.py`` to tuning:
+
+.. code:: bash
+
+	python run_hyper.py --model=[model_name] --dataset=[dataset_name] --config_files=[config_files_path] --params_file=hyper.test
+
+For more details about Parameter Tuning, refer to :doc:`../../../user_guide/usage/parameter_tuning`.
+
+
+If you want to change parameters, dataset or evaluation settings, take a look at
+
+- :doc:`../../../user_guide/config_settings`
+- :doc:`../../../user_guide/data_intro`
+- :doc:`../../../user_guide/train_eval_intro`
+- :doc:`../../../user_guide/usage`
diff --git a/docs/source/user_guide/model/sequential/sine.rst b/docs/source/user_guide/model/sequential/sine.rst
new file mode 100644
index 000000000..8f6b38e7c
--- /dev/null
+++ b/docs/source/user_guide/model/sequential/sine.rst
@@ -0,0 +1,97 @@
+SINE
+=================
+
+Introduction
+---------------------
+
+`[paper] <https://doi.org/10.1145/3437963.3441811>`_
+
+**Title:** Sparse-Interest Network for Sequential Recommendation
+
+**Authors:** Qiaoyu Tan, Jianwei Zhang, Jiangchao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang and Xia Hu
+
+**Abstract:**  Recent methods in sequential recommendation focus on learning an overall embedding vector 
+from a user's behavior sequence for the next-item recommendation. However, from empirical 
+analysis, we discovered that a user's behavior sequence often contains multiple conceptually 
+distinct items, while a unified embedding vector is primarily affected by one's most recent 
+frequent actions. Thus, it may fail to infer the next preferred item if conceptually similar 
+items are not dominant in recent interactions. To this end, an alternative solution is to 
+represent each user with multiple embedding vectors encoding different aspects of the user's 
+intentions. Nevertheless, recent work on multi-interest embedding usually considers a small 
+number of concepts discovered via clustering, which may not be comparable to the large pool of 
+item categories in real systems. It is a non-trivial task to effectively model a large number 
+of diverse conceptual prototypes, as items are often not conceptually well clustered in fine 
+granularity. Besides, an individual usually interacts with only a sparse set of concepts. In 
+light of this, we propose a novel Sparse Interest NEtwork (SINE) for sequential recommendation. 
+Our sparse-interest module can adaptively infer a sparse set of concepts for each user from the 
+large concept pool and output multiple embeddings accordingly. Given multiple interest embeddings, 
+we develop an interest aggregation module to actively predict the user's current intention and 
+then use it to explicitly model multiple interests for next-item prediction. Empirical results 
+on several public benchmark datasets and one large-scale industrial dataset demonstrate that SINE 
+can achieve substantial improvement over state-of-the-art methods.
+
+.. image:: ../../../asset/sine.png
+    :width: 500
+    :align: center
+
+Running with RecBole
+-------------------------
+
+**Model Hyper-Parameters:**
+
+- ``embedding_size (int)`` : The input and output hidden size. Defaults to ``64``.
+- ``layer_norm_eps (float)``: The value added to the denominator for numerical stability. Defaults to ``1e-12``.
+- ``prototype_size (int)``: The size of latent prototypes. Defaults to ``50``.
+- ``interest_size (int)`` : The number of intentions. Defaults to ``3``.
+- ``tau_ratio (float)``: The tau value for temperature tuning. Defaults to ``0.1``.
+- ``reg_loss_ratio (float)``: The L2 regularization weight. Defaults to ``0.5``.
+- ``loss_type (str)`` : The type of loss function. If ``loss_type='CE'``, the training task is regarded as a multi-classification task and the target item is the ground truth. In this way, negative sampling is not needed. If ``loss_type='BPR'``, the training task will be optimized in the pair-wise way, which maximizes the difference between positive item and negative item. In this way, negative sampling is necessary, such as setting ``--neg_sampling="{'uniform': 1}"``. Defaults to ``'CE'``. Range in ``['BPR', 'CE']``.
+
+
+**A Running Example:**
+
+Write the following code to a python file, such as `run.py`
+
+.. code:: python
+
+   from recbole.quick_start import run_recbole
+
+   parameter_dict = {
+      'neg_sampling': None,
+   }
+   run_recbole(model='SINE', dataset='ml-100k', config_dict=parameter_dict)
+
+And then:
+
+.. code:: bash
+
+   python run.py
+
+Tuning Hyper Parameters
+-------------------------
+
+If you want to use ``HyperTuning`` to tune hyper parameters of this model, you can copy the following settings and name it as ``hyper.test``.
+
+.. code:: bash
+
+   learning_rate choice [0.01,0.005,0.001,0.0005,0.0001]
+   prototype_size choice [50,100,500,1000,2000,5000]
+   interest_size choice [4,8,12,16]
+
+Note that we just provide these hyper parameter ranges for reference only, and we can not guarantee that they are the optimal range of this model.
+
+Then, with the source code of RecBole (you can download it from GitHub), you can run the ``run_hyper.py`` to tuning:
+
+.. code:: bash
+
+	python run_hyper.py --model=[model_name] --dataset=[dataset_name] --config_files=[config_files_path] --params_file=hyper.test
+
+For more details about Parameter Tuning, refer to :doc:`../../../user_guide/usage/parameter_tuning`.
+
+
+If you want to change parameters, dataset or evaluation settings, take a look at
+
+- :doc:`../../../user_guide/config_settings`
+- :doc:`../../../user_guide/data_intro`
+- :doc:`../../../user_guide/train_eval_intro`
+- :doc:`../../../user_guide/usage`
\ No newline at end of file
diff --git a/docs/source/user_guide/model_intro.rst b/docs/source/user_guide/model_intro.rst
index ef254b6fd..31748e574 100644
--- a/docs/source/user_guide/model_intro.rst
+++ b/docs/source/user_guide/model_intro.rst
@@ -1,6 +1,6 @@
 Model Introduction
 =====================
-We implement 73 recommendation models covering general recommendation, sequential recommendation,
+We implement 77 recommendation models covering general recommendation, sequential recommendation,
 context-aware recommendation and knowledge-based recommendation. A brief introduction to these models are as follows:
 
 
@@ -37,6 +37,8 @@ task of top-n recommendation. All the collaborative filter(CF) based models are
    model/general/recvae
    model/general/ease
    model/general/slimelastic
+   model/general/sgl
+   model/general/admmslim
    model/general/nceplrec
 
 
@@ -101,7 +103,8 @@ the sequential data. The models of session-based recommendation are also include
    model/sequential/hgn
    model/sequential/hrm
    model/sequential/npe
-
+   model/sequential/lightsans
+   model/sequential/sine
 
 
 Knowledge-based Recommendation
diff --git a/recbole/config/configurator.py b/recbole/config/configurator.py
index c7307818c..e61853778 100644
--- a/recbole/config/configurator.py
+++ b/recbole/config/configurator.py
@@ -354,8 +354,6 @@ def _set_train_neg_sample_args(self):
         else:
             if not isinstance(neg_sampling, dict):
                 raise ValueError(f"neg_sampling:[{neg_sampling}] should be a dict.")
-            if len(neg_sampling) > 1:
-                raise ValueError(f"the len of neg_sampling [{neg_sampling}] should be 1.")
 
             distribution = list(neg_sampling.keys())[0]
             sample_num = neg_sampling[distribution]
@@ -363,10 +361,15 @@ def _set_train_neg_sample_args(self):
                 raise ValueError(f"The distribution [{distribution}] of neg_sampling "
                                  f"should in ['uniform', 'popularity']")
 
+            dynamic = 'none'
+            if 'dynamic' in neg_sampling.keys():
+                dynamic = neg_sampling['dynamic']
+
             self.final_config_dict['train_neg_sample_args'] = {
                 'strategy': 'by',
                 'by': sample_num,
-                'distribution': distribution
+                'distribution': distribution,
+                'dynamic': dynamic
             }
 
     def _set_eval_neg_sample_args(self):
diff --git a/recbole/data/dataloader/abstract_dataloader.py b/recbole/data/dataloader/abstract_dataloader.py
index 84a762182..91b6d79d6 100644
--- a/recbole/data/dataloader/abstract_dataloader.py
+++ b/recbole/data/dataloader/abstract_dataloader.py
@@ -13,6 +13,7 @@
 """
 
 import math
+import copy
 from logging import getLogger
 
 import torch
@@ -45,7 +46,7 @@ def __init__(self, config, dataset, sampler, shuffle=False):
         self.logger = getLogger()
         self.dataset = dataset
         self.sampler = sampler
-        self.batch_size = self.step = None
+        self.batch_size = self.step = self.model = None
         self.shuffle = shuffle
         self.pr = 0
         self._init_batch_size_and_step()
@@ -54,6 +55,15 @@ def _init_batch_size_and_step(self):
         """Initializing :attr:`step` and :attr:`batch_size`."""
         raise NotImplementedError('Method [init_batch_size_and_step] should be implemented')
 
+    def update_config(self, config):
+        """Update configure of dataloader, such as :attr:`batch_size`, :attr:`step` etc.
+
+        Args:
+            config (Config): The new config of dataloader.
+        """
+        self.config = config
+        self._init_batch_size_and_step()
+
     def __len__(self):
         return math.ceil(self.pr_end / self.step)
 
@@ -145,7 +155,23 @@ def _set_neg_sample_args(self, config, dataset, dl_format, neg_sample_args):
             raise ValueError(f'`neg_sample_args` [{self.neg_sample_args["strategy"]}] is not supported!')
 
     def _neg_sampling(self, inter_feat):
-        if self.neg_sample_args['strategy'] == 'by':
+        if 'dynamic' in self.neg_sample_args.keys() and self.neg_sample_args['dynamic'] != 'none':
+            candidate_num = self.neg_sample_args['dynamic']
+            user_ids = inter_feat[self.uid_field].numpy()
+            item_ids = inter_feat[self.iid_field].numpy()
+            neg_candidate_ids = self.sampler.sample_by_user_ids(user_ids, item_ids, self.neg_sample_num * candidate_num)
+            self.model.eval()
+            interaction = copy.deepcopy(inter_feat).to(self.model.device)
+            interaction = interaction.repeat(self.neg_sample_num * candidate_num)
+            neg_item_feat = Interaction({self.iid_field: neg_candidate_ids.to(self.model.device)})
+            interaction.update(neg_item_feat)
+            scores = self.model.predict(interaction).reshape(candidate_num, -1)
+            indices = torch.max(scores, dim=0)[1].detach()
+            neg_candidate_ids = neg_candidate_ids.reshape(candidate_num, -1)
+            neg_item_ids = neg_candidate_ids[indices, [i for i in range(neg_candidate_ids.shape[1])]].view(-1)
+            self.model.train()
+            return self.sampling_func(inter_feat, neg_item_ids)
+        elif self.neg_sample_args['strategy'] == 'by':
             user_ids = inter_feat[self.uid_field].numpy()
             item_ids = inter_feat[self.iid_field].numpy()
             neg_item_ids = self.sampler.sample_by_user_ids(user_ids, item_ids, self.neg_sample_num)
@@ -170,3 +196,6 @@ def _neg_sample_by_point_wise_sampling(self, inter_feat, neg_item_ids):
         labels[:pos_inter_num] = 1.0
         new_data.update(Interaction({self.label_field: labels}))
         return new_data
+
+    def get_model(self, model):
+        self.model = model
diff --git a/recbole/data/dataloader/general_dataloader.py b/recbole/data/dataloader/general_dataloader.py
index aaefdb6e5..dac73d76c 100644
--- a/recbole/data/dataloader/general_dataloader.py
+++ b/recbole/data/dataloader/general_dataloader.py
@@ -48,6 +48,10 @@ def _init_batch_size_and_step(self):
             self.step = batch_size
             self.set_batch_size(batch_size)
 
+    def update_config(self, config):
+        self._set_neg_sample_args(config, self.dataset, config['MODEL_INPUT_TYPE'], config['train_neg_sample_args'])
+        super().update_config(config)
+
     @property
     def pr_end(self):
         return len(self.dataset)
@@ -112,6 +116,10 @@ def _init_batch_size_and_step(self):
             self.step = batch_size
             self.set_batch_size(batch_size)
 
+    def update_config(self, config):
+        self._set_neg_sample_args(config, self.dataset, InputType.POINTWISE, config['eval_neg_sample_args'])
+        super().update_config(config)
+
     @property
     def pr_end(self):
         if self.neg_sample_args['strategy'] == 'by':
diff --git a/recbole/data/dataloader/knowledge_dataloader.py b/recbole/data/dataloader/knowledge_dataloader.py
index bcb6bc4b9..4994472e2 100644
--- a/recbole/data/dataloader/knowledge_dataloader.py
+++ b/recbole/data/dataloader/knowledge_dataloader.py
@@ -116,6 +116,10 @@ def __init__(self, config, dataset, sampler, kg_sampler, shuffle=False):
     def _init_batch_size_and_step(self):
         pass
 
+    def update_config(self, config):
+        self.general_dataloader.update_config(config)
+        self.kg_dataloader.update_config(config)
+
     def __iter__(self):
         if self.state is None:
             raise ValueError(
@@ -178,3 +182,8 @@ def set_mode(self, state):
         if state not in set(KGDataLoaderState):
             raise NotImplementedError(f'Kg data loader has no state named [{self.state}].')
         self.state = state
+
+    def get_model(self, model):
+        """Let the general_dataloader get the model, used for dynamic sampling.
+        """
+        self.general_dataloader.get_model(model)
\ No newline at end of file
diff --git a/recbole/data/dataset/dataset.py b/recbole/data/dataset/dataset.py
index b430a249d..30c06156a 100644
--- a/recbole/data/dataset/dataset.py
+++ b/recbole/data/dataset/dataset.py
@@ -26,11 +26,11 @@
 from scipy.sparse import coo_matrix
 
 from recbole.data.interaction import Interaction
-from recbole.utils import FeatureSource, FeatureType, get_local_time, set_color
+from recbole.utils import FeatureSource, FeatureType, get_local_time, set_color, ensure_dir
 from recbole.utils.url import decide_download, download_url, extract_zip, makedirs, rename_atomic_files
 
 
-class Dataset(object):
+class Dataset:
     """:class:`Dataset` stores the original dataset in memory.
     It provides many useful functions for data preprocessing, such as k-core data filtering and missing value
     imputation. Features are stored as :class:`pandas.DataFrame` inside :class:`~recbole.data.dataset.dataset.Dataset`.
@@ -305,6 +305,8 @@ def _load_user_or_item_feat(self, token, dataset_path, source, field_name):
             :obj:`~recbole.utils.enum_type.FeatureSource.ITEM_ID`
         """
         feat_path = os.path.join(dataset_path, f'{token}.{source.value}')
+        field = getattr(self, field_name, None)
+
         if os.path.isfile(feat_path):
             feat = self._load_feat(feat_path, source)
             self.logger.debug(f'[{source.value}] feature loaded successfully from [{feat_path}].')
@@ -312,11 +314,12 @@ def _load_user_or_item_feat(self, token, dataset_path, source, field_name):
             feat = None
             self.logger.debug(f'[{feat_path}] not found, [{source.value}] features are not loaded.')
 
-        field = getattr(self, field_name, None)
         if feat is not None and field is None:
             raise ValueError(f'{field_name} must be exist if {source.value}_feat exist.')
         if feat is not None and field not in feat:
             raise ValueError(f'{field_name} must be loaded if {source.value}_feat is loaded.')
+        if feat is not None:
+            feat.drop_duplicates(subset=[field], keep='first', inplace=True)
 
         if field in self.field2source:
             self.field2source[field] = FeatureSource(source.value + '_id')
@@ -433,7 +436,7 @@ def _load_feat(self, filepath, source):
             return None
 
         df = pd.read_csv(
-            filepath, delimiter=self.config['field_separator'], usecols=usecols, dtype=dtype, encoding=encoding
+            filepath, delimiter=field_separator, usecols=usecols, dtype=dtype, encoding=encoding, engine='python'
         )
         df.columns = columns
 
@@ -681,6 +684,9 @@ def _filter_by_inter_num(self):
         user_inter_num_interval = self._parse_intervals_str(self.config['user_inter_num_interval'])
         item_inter_num_interval = self._parse_intervals_str(self.config['item_inter_num_interval'])
 
+        if user_inter_num_interval is None and item_inter_num_interval is None:
+            return
+
         user_inter_num = Counter(self.inter_feat[self.uid_field].values) if user_inter_num_interval else Counter()
         item_inter_num = Counter(self.inter_feat[self.iid_field].values) if item_inter_num_interval else Counter()
 
@@ -798,18 +804,20 @@ def _filter_by_field_value(self):
         """Filter features according to its values.
         """
 
-        val_intervals = [] if self.config['val_interval'] is None else self.config['val_interval']
+        val_intervals = {} if self.config['val_interval'] is None else self.config['val_interval']
         self.logger.debug(set_color('drop_by_value', 'blue') + f': val={val_intervals}')
 
-        for field in val_intervals:
+        for field, interval in val_intervals.items():
             if field not in self.field2type:
                 raise ValueError(f'Field [{field}] not defined in dataset.')
-            if self.field2type[field] not in {FeatureType.FLOAT, FeatureType.FLOAT_SEQ}:
-                raise ValueError(f'Field [{field}] is not float-like field in dataset, which can\'t be filter.')
 
-            field_val_interval = self._parse_intervals_str(val_intervals[field])
-            for feat in self.field2feats(field):
-                feat.drop(feat.index[~self._within_intervals(feat[field].values, field_val_interval)], inplace=True)
+            if self.field2type[field] in {FeatureType.FLOAT, FeatureType.FLOAT_SEQ}:
+                field_val_interval = self._parse_intervals_str(interval)
+                for feat in self.field2feats(field):
+                    feat.drop(feat.index[~self._within_intervals(feat[field].values, field_val_interval)], inplace=True)
+            else:  # token-like field
+                for feat in self.field2feats(field):
+                    feat.drop(feat.index[~feat[field].isin(interval)], inplace=True)
 
     def _reset_index(self):
         """Reset index for all feats in :attr:`feat_name_list`.
@@ -1509,7 +1517,7 @@ def save(self):
         """Saving this :class:`Dataset` object to :attr:`config['checkpoint_dir']`.
         """
         save_dir = self.config['checkpoint_dir']
-        ensure_dir(self.config['checkpoint_dir'])
+        ensure_dir(save_dir)
         file = os.path.join(save_dir, f'{self.config["dataset"]}-dataset.pth')
         self.logger.info(set_color('Saving filtered dataset into ', 'pink') + f'[{file}]')
         with open(file, 'wb') as f:
diff --git a/recbole/data/dataset/sequential_dataset.py b/recbole/data/dataset/sequential_dataset.py
index 82146b123..ac749a780 100644
--- a/recbole/data/dataset/sequential_dataset.py
+++ b/recbole/data/dataset/sequential_dataset.py
@@ -125,9 +125,7 @@ def data_augmentation(self):
                 list_field = getattr(self, f'{field}_list_field')
                 list_len = self.field2seqlen[list_field]
                 shape = (new_length, list_len) if isinstance(list_len, int) else (new_length,) + list_len
-                list_ftype = self.field2type[list_field]
-                dtype = torch.int64 if list_ftype in [FeatureType.TOKEN, FeatureType.TOKEN_SEQ] else torch.float64
-                new_dict[list_field] = torch.zeros(shape, dtype=dtype)
+                new_dict[list_field] = torch.zeros(shape, dtype=self.inter_feat[field].dtype)
 
                 value = self.inter_feat[field]
                 for i, (index, length) in enumerate(zip(item_list_index, item_list_length)):
diff --git a/recbole/data/interaction.py b/recbole/data/interaction.py
index 7a557888d..054c6b3ff 100644
--- a/recbole/data/interaction.py
+++ b/recbole/data/interaction.py
@@ -135,6 +135,16 @@ def __getitem__(self, index):
                 ret[k] = self.interaction[k][index]
             return Interaction(ret)
 
+    def __setitem__(self, key, value):
+        if not isinstance(key, str):
+            raise KeyError(f'{type(key)} object does not support item assigment')
+        self.interaction[key] = value
+
+    def __delitem__(self, key):
+        if key not in self.interaction:
+            raise KeyError(f'{type(key)} object does not in this interaction')
+        del self.interaction[key]
+
     def __contains__(self, item):
         return item in self.interaction
 
diff --git a/recbole/data/utils.py b/recbole/data/utils.py
index 8ff65afdd..1ad22ae23 100644
--- a/recbole/data/utils.py
+++ b/recbole/data/utils.py
@@ -20,10 +20,14 @@
 from recbole.data.dataloader import *
 from recbole.sampler import KGSampler, Sampler, RepeatableSampler
 from recbole.utils import ModelType, ensure_dir, get_local_time, set_color
+from recbole.utils.argument_list import dataset_arguments
 
 
 def create_dataset(config):
     """Create dataset according to :attr:`config['model']` and :attr:`config['MODEL_TYPE']`.
+    If :attr:`config['dataset_save_path']` file exists and
+    its :attr:`config` of dataset is equal to current :attr:`config` of dataset.
+    It will return the saved dataset in :attr:`config['dataset_save_path']`.
 
     Args:
         config (Config): An instance object of Config, used to record parameter information.
@@ -33,21 +37,38 @@ def create_dataset(config):
     """
     dataset_module = importlib.import_module('recbole.data.dataset')
     if hasattr(dataset_module, config['model'] + 'Dataset'):
-        return getattr(dataset_module, config['model'] + 'Dataset')(config)
+        dataset_class = getattr(dataset_module, config['model'] + 'Dataset')
     else:
         model_type = config['MODEL_TYPE']
-        if model_type == ModelType.SEQUENTIAL:
-            from .dataset import SequentialDataset
-            return SequentialDataset(config)
-        elif model_type == ModelType.KNOWLEDGE:
-            from .dataset import KnowledgeBasedDataset
-            return KnowledgeBasedDataset(config)
-        elif model_type == ModelType.DECISIONTREE:
-            from .dataset import DecisionTreeDataset
-            return DecisionTreeDataset(config)
-        else:
-            from .dataset import Dataset
-            return Dataset(config)
+        type2class = {
+            ModelType.GENERAL: 'Dataset',
+            ModelType.SEQUENTIAL: 'SequentialDataset',
+            ModelType.CONTEXT: 'Dataset',
+            ModelType.KNOWLEDGE: 'KnowledgeBasedDataset',
+            ModelType.TRADITIONAL: 'Dataset',
+            ModelType.DECISIONTREE: 'Dataset',
+        }
+        dataset_class = getattr(dataset_module, type2class[model_type])
+
+    default_file = os.path.join(config['checkpoint_dir'], f'{config["dataset"]}-{dataset_class.__name__}.pth')
+    file = config['dataset_save_path'] or default_file
+    if os.path.exists(file):
+        with open(file, 'rb') as f:
+            dataset = pickle.load(f)
+        dataset_args_unchanged = True
+        for arg in dataset_arguments + ['seed', 'repeatable']:
+            if config[arg] != dataset.config[arg]:
+                dataset_args_unchanged = False
+                break
+        if dataset_args_unchanged:
+            logger = getLogger()
+            logger.info(set_color('Load filtered dataset from', 'pink') + f': [{file}]')
+            return dataset
+
+    dataset = dataset_class(config)
+    if config['save_dataset']:
+        dataset.save()
+    return dataset
 
 
 def save_split_dataloaders(config, dataloaders):
@@ -57,37 +78,53 @@ def save_split_dataloaders(config, dataloaders):
         config (Config): An instance object of Config, used to record parameter information.
         dataloaders (tuple of AbstractDataLoader): The split dataloaders.
     """
+    ensure_dir(config['checkpoint_dir'])
     save_path = config['checkpoint_dir']
     saved_dataloaders_file = f'{config["dataset"]}-for-{config["model"]}-dataloader.pth'
     file_path = os.path.join(save_path, saved_dataloaders_file)
     logger = getLogger()
-    logger.info(set_color('Saved split dataloaders', 'blue') + f': {file_path}')
+    logger.info(set_color('Saving split dataloaders into', 'pink') + f': [{file_path}]')
     with open(file_path, 'wb') as f:
         pickle.dump(dataloaders, f)
 
 
-def load_split_dataloaders(saved_dataloaders_file):
-    """Load split dataloaders.
+def load_split_dataloaders(config):
+    """Load split dataloaders if saved dataloaders exist and
+    their :attr:`config` of dataset are the same as current :attr:`config` of dataset.
 
     Args:
-        saved_dataloaders_file (str): The path of split dataloaders.
+        config (Config): An instance object of Config, used to record parameter information.
 
     Returns:
-        dataloaders (tuple of AbstractDataLoader): The split dataloaders.
+        dataloaders (tuple of AbstractDataLoader or None): The split dataloaders.
     """
-    with open(saved_dataloaders_file, 'rb') as f:
-        dataloaders = pickle.load(f)
-    return dataloaders
 
+    default_file = os.path.join(config['checkpoint_dir'], f'{config["dataset"]}-for-{config["model"]}-dataloader.pth')
+    dataloaders_save_path = config['dataloaders_save_path'] or default_file
+    if not os.path.exists(dataloaders_save_path):
+        return None
+    with open(dataloaders_save_path, 'rb') as f:
+        train_data, valid_data, test_data = pickle.load(f)
+    for arg in dataset_arguments + ['seed', 'repeatable', 'eval_args']:
+        if config[arg] != train_data.config[arg]:
+            return None
+    train_data.update_config(config)
+    valid_data.update_config(config)
+    test_data.update_config(config)
+    logger = getLogger()
+    logger.info(set_color('Load split dataloaders from', 'pink') + f': [{dataloaders_save_path}]')
+    return train_data, valid_data, test_data
 
-def data_preparation(config, dataset, save=False):
+
+def data_preparation(config, dataset):
     """Split the dataset by :attr:`config['eval_args']` and create training, validation and test dataloader.
 
+    Note:
+        If we can load split dataloaders by :meth:`load_split_dataloaders`, we will not create new split dataloaders.
+
     Args:
         config (Config): An instance object of Config, used to record parameter information.
         dataset (Dataset): An instance object of Dataset, which contains all interaction records.
-        save (bool, optional): If ``True``, it will call :func:`save_datasets` to save split dataset.
-            Defaults to ``False``.
 
     Returns:
         tuple:
@@ -95,21 +132,28 @@ def data_preparation(config, dataset, save=False):
             - valid_data (AbstractDataLoader): The dataloader for validation.
             - test_data (AbstractDataLoader): The dataloader for testing.
     """
-    model_type = config['MODEL_TYPE']
-    built_datasets = dataset.build()
-    logger = getLogger()
+    dataloaders = load_split_dataloaders(config)
+    if dataloaders is not None:
+        train_data, valid_data, test_data = dataloaders
+    else:
+        model_type = config['MODEL_TYPE']
+        built_datasets = dataset.build()
 
-    train_dataset, valid_dataset, test_dataset = built_datasets
-    train_sampler, valid_sampler, test_sampler = create_samplers(config, dataset, built_datasets)
+        train_dataset, valid_dataset, test_dataset = built_datasets
+        train_sampler, valid_sampler, test_sampler = create_samplers(config, dataset, built_datasets)
 
-    if model_type != ModelType.KNOWLEDGE:
-        train_data = get_dataloader(config, 'train')(config, train_dataset, train_sampler, shuffle=True)
-    else:
-        kg_sampler = KGSampler(dataset, config['train_neg_sample_args']['distribution'])
-        train_data = get_dataloader(config, 'train')(config, train_dataset, train_sampler, kg_sampler, shuffle=True)
+        if model_type != ModelType.KNOWLEDGE:
+            train_data = get_dataloader(config, 'train')(config, train_dataset, train_sampler, shuffle=True)
+        else:
+            kg_sampler = KGSampler(dataset, config['train_neg_sample_args']['distribution'])
+            train_data = get_dataloader(config, 'train')(config, train_dataset, train_sampler, kg_sampler, shuffle=True)
+
+        valid_data = get_dataloader(config, 'evaluation')(config, valid_dataset, valid_sampler, shuffle=False)
+        test_data = get_dataloader(config, 'evaluation')(config, test_dataset, test_sampler, shuffle=False)
+        if config['save_dataloaders']:
+            save_split_dataloaders(config, dataloaders=(train_data, valid_data, test_data))
 
-    valid_data = get_dataloader(config, 'evaluation')(config, valid_dataset, valid_sampler, shuffle=False)
-    test_data = get_dataloader(config, 'evaluation')(config, test_dataset, test_sampler, shuffle=False)
+    logger = getLogger()
     logger.info(
         set_color('[Training]: ', 'pink') + set_color('train_batch_size', 'cyan') + ' = ' +
         set_color(f'[{config["train_batch_size"]}]', 'yellow') + set_color(' negative sampling', 'cyan') + ': ' +
@@ -120,9 +164,6 @@ def data_preparation(config, dataset, save=False):
         set_color(f'[{config["eval_batch_size"]}]', 'yellow') + set_color(' eval_args', 'cyan') + ': ' +
         set_color(f'[{config["eval_args"]}]', 'yellow')
     )
-    if save:
-        save_split_dataloaders(config, dataloaders=(train_data, valid_data, test_data))
-
     return train_data, valid_data, test_data
 
 
diff --git a/recbole/evaluator/base_metric.py b/recbole/evaluator/base_metric.py
index 695cd5be8..b075beb2d 100644
--- a/recbole/evaluator/base_metric.py
+++ b/recbole/evaluator/base_metric.py
@@ -60,7 +60,7 @@ def used_info(self, dataobject):
         """
         rec_mat = dataobject.get('rec.topk')
         topk_idx, pos_len_list = torch.split(rec_mat, [max(self.topk), 1], dim=1)
-        return rec_mat.to(torch.bool).numpy(), pos_len_list.squeeze(-1).numpy()
+        return topk_idx.to(torch.bool).numpy(), pos_len_list.squeeze(-1).numpy()
 
     def topk_result(self, metric, value):
         """Match the metric value to the `k` and put them in `dictionary` form.
diff --git a/recbole/evaluator/evaluator.py b/recbole/evaluator/evaluator.py
index d83dda333..215c8eae6 100644
--- a/recbole/evaluator/evaluator.py
+++ b/recbole/evaluator/evaluator.py
@@ -10,6 +10,7 @@
 
 from recbole.evaluator.register import metrics_dict
 from recbole.evaluator.collector import DataStruct
+from collections import OrderedDict
 
 
 class Evaluator(object):
@@ -31,10 +32,10 @@ def evaluate(self, dataobject: DataStruct):
             dataobject (DataStruct): It contains all the information needed for metrics.
 
         Returns:
-            dict: such as ``{'hit@20': 0.3824, 'recall@20': 0.0527, 'hit@10': 0.3153, 'recall@10': 0.0329, 'gauc': 0.9236}``
+            collections.OrderedDict: such as ``{'hit@20': 0.3824, 'recall@20': 0.0527, 'hit@10': 0.3153, 'recall@10': 0.0329, 'gauc': 0.9236}``
 
         """
-        result_dict = {}
+        result_dict = OrderedDict()
         for metric in self.metrics:
             metric_val = self.metric_class[metric].calculate_metric(dataobject)
             result_dict.update(metric_val)
diff --git a/recbole/model/abstract_recommender.py b/recbole/model/abstract_recommender.py
index 2aaae77a6..547aac0e7 100644
--- a/recbole/model/abstract_recommender.py
+++ b/recbole/model/abstract_recommender.py
@@ -124,12 +124,24 @@ def __init__(self, config, dataset):
         self.max_seq_length = config['MAX_ITEM_LIST_LENGTH']
         self.n_items = dataset.num(self.ITEM_ID)
 
+        # load parameters info
+        self.device = config['device']
+
     def gather_indexes(self, output, gather_index):
         """Gathers the vectors at the specific positions over a minibatch"""
         gather_index = gather_index.view(-1, 1, 1).expand(-1, -1, output.shape[-1])
         output_tensor = output.gather(dim=1, index=gather_index)
         return output_tensor.squeeze(1)
 
+    def get_attention_mask(self, item_seq, bidirectional=False):
+        """Generate left-to-right uni-directional or bidirectional attention mask for multi-head attention."""
+        attention_mask = (item_seq != 0)
+        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.bool
+        if not bidirectional:
+            extended_attention_mask = torch.tril(extended_attention_mask.expand((-1, -1, item_seq.size(-1), -1)))
+        extended_attention_mask = torch.where(extended_attention_mask, 0., -10000.)
+        return extended_attention_mask
+
 
 class KnowledgeRecommender(AbstractRecommender):
     """This is a abstract knowledge-based recommender. All the knowledge-based model should implement this class.
diff --git a/recbole/model/general_recommender/__init__.py b/recbole/model/general_recommender/__init__.py
index ea36ffbf0..b8cdd23c4 100644
--- a/recbole/model/general_recommender/__init__.py
+++ b/recbole/model/general_recommender/__init__.py
@@ -24,3 +24,6 @@
 from recbole.model.general_recommender.slimelastic import SLIMElastic
 from recbole.model.general_recommender.spectralcf import SpectralCF
 from recbole.model.general_recommender.nceplrec import NCEPLRec
+from recbole.model.general_recommender.sgl import SGL
+from recbole.model.general_recommender.admmslim import ADMMSLIM
+
diff --git a/recbole/model/general_recommender/admmslim.py b/recbole/model/general_recommender/admmslim.py
new file mode 100644
index 000000000..d8242904e
--- /dev/null
+++ b/recbole/model/general_recommender/admmslim.py
@@ -0,0 +1,117 @@
+# @Time   : 2021/01/09
+# @Author : Deklan Webster
+
+r"""
+ADMMSLIM
+################################################
+Reference:
+    Steck et al. ADMM SLIM: Sparse Recommendations for Many Users. https://doi.org/10.1145/3336191.3371774
+
+"""
+
+from recbole.utils.enum_type import ModelType
+import numpy as np
+import scipy.sparse as sp
+import torch
+
+from recbole.utils import InputType
+from recbole.model.abstract_recommender import GeneralRecommender
+
+
+def soft_threshold(x, threshold):
+    return (np.abs(x) > threshold) * (np.abs(x) - threshold) * np.sign(x)
+
+
+def zero_mean_columns(a):
+    return a - np.mean(a, axis=0)
+
+
+def add_noise(t, mag=1e-5):
+    return t + mag * torch.rand(t.shape)
+
+
+class ADMMSLIM(GeneralRecommender):
+    input_type = InputType.POINTWISE
+    type = ModelType.TRADITIONAL
+
+    def __init__(self, config, dataset):
+        super().__init__(config, dataset)
+
+        # need at least one param
+        self.dummy_param = torch.nn.Parameter(torch.zeros(1))
+
+        X = dataset.inter_matrix(form='csr').astype(np.float32)
+
+        num_users, num_items = X.shape
+
+        lambda1 = config['lambda1']
+        lambda2 = config['lambda2']
+        alpha = config['alpha']
+        rho = config['rho']
+        k = config['k']
+        positive_only = config['positive_only']
+        self.center_columns = config['center_columns']
+        self.item_means = X.mean(axis=0).getA1()
+
+        if self.center_columns:
+            zero_mean_X = X.toarray() - self.item_means
+            G = (zero_mean_X.T @ zero_mean_X)
+            # large memory cost because we need to make X dense to subtract mean, delete asap
+            del zero_mean_X
+        else:
+            G = (X.T @ X).toarray()
+
+        diag = lambda2 * np.diag(np.power(self.item_means, alpha)) + \
+            rho * np.identity(num_items)
+
+        P = np.linalg.inv(G + diag).astype(np.float32)
+        B_aux = (P @ G).astype(np.float32)
+        # initialize
+        Gamma = np.zeros_like(G, dtype=np.float32)
+        C = np.zeros_like(G, dtype=np.float32)
+
+        del diag, G
+        # fixed number of iterations
+        for _ in range(k):
+            B_tilde = B_aux + P @ (rho * C - Gamma)
+            gamma = np.diag(B_tilde) / (np.diag(P) + 1e-7)
+            B = B_tilde - P * gamma
+            C = soft_threshold(B + Gamma / rho, lambda1 / rho)
+            if positive_only:
+                C = (C > 0) * C
+            Gamma += rho * (B - C)
+        # torch doesn't support sparse tensor slicing, so will do everything with np/scipy
+        self.item_similarity = C
+        self.interaction_matrix = X
+
+    def forward(self):
+        pass
+
+    def calculate_loss(self, interaction):
+        return torch.nn.Parameter(torch.zeros(1))
+
+    def predict(self, interaction):
+        user = interaction[self.USER_ID].cpu().numpy()
+        item = interaction[self.ITEM_ID].cpu().numpy()
+
+        user_interactions = self.interaction_matrix[user, :].toarray()
+
+        if self.center_columns:
+            r = (((user_interactions - self.item_means) *
+                  self.item_similarity[:, item].T).sum(axis=1)).flatten() + self.item_means[item]
+        else:
+            r = (user_interactions * self.item_similarity[:, item].T).sum(axis=1).flatten()
+
+        return add_noise(torch.from_numpy(r))
+
+    def full_sort_predict(self, interaction):
+        user = interaction[self.USER_ID].cpu().numpy()
+
+        user_interactions = self.interaction_matrix[user, :].toarray()
+
+        if self.center_columns:
+            r = ((user_interactions - self.item_means) @ self.item_similarity + self.item_means).flatten()
+        else:
+            r = (user_interactions @ self.item_similarity).flatten()
+
+        return add_noise(torch.from_numpy(r))
diff --git a/recbole/model/general_recommender/sgl.py b/recbole/model/general_recommender/sgl.py
new file mode 100644
index 000000000..4c7fea69b
--- /dev/null
+++ b/recbole/model/general_recommender/sgl.py
@@ -0,0 +1,281 @@
+# -*- coding: utf-8 -*-
+# @Time   : 2021/10/12
+# @Author : Tian Zhen
+# @Email  : chenyuwuxinn@gmail.com
+
+r"""
+SGL
+################################################
+Reference:
+    Jiancan Wu et al. "SGL: Self-supervised Graph Learning for Recommendation" in SIGIR 2021.
+
+Reference code:
+    https://github.com/wujcan/SGL
+"""
+
+import numpy as np
+import scipy.sparse as sp
+import torch
+from recbole.model.abstract_recommender import GeneralRecommender
+from recbole.model.init import xavier_uniform_initialization
+from recbole.model.loss import BPRLoss, EmbLoss
+from recbole.utils import InputType
+import torch.nn.functional as F
+
+
+class SGL(GeneralRecommender):
+    r"""SGL is a GCN-based recommender model.
+
+    SGL supplements the classical supervised task of recommendation with an auxiliary 
+    self supervised task, which reinforces node representation learning via self-
+    discrimination.Specifically,SGL generates multiple views of a node, maximizing the
+    agreement between different views of the same node compared to that of other nodes.
+    SGL devises three operators to generate the views — node dropout, edge dropout, and 
+    random walk — that change the graph structure in different manners.
+
+    We implement the model following the original author with a pairwise training mode.
+    """
+    input_type = InputType.PAIRWISE
+
+    def __init__(self, config, dataset):
+        super(SGL, self).__init__(config, dataset)
+        self._user = dataset.inter_feat[dataset.uid_field]
+        self._item = dataset.inter_feat[dataset.iid_field]
+        self.embed_dim = config["embedding_size"]
+        self.n_layers = int(config["n_layers"])
+        self.type = config["type"]
+        self.drop_ratio = config["drop_ratio"]
+        self.ssl_tau = config["ssl_tau"]
+        self.reg_weight = config["reg_weight"]
+        self.ssl_weight = config["ssl_weight"]
+        self.user_embedding = torch.nn.Embedding(self.n_users, self.embed_dim, device=self.device)
+        self.item_embedding = torch.nn.Embedding(self.n_items, self.embed_dim, device=self.device)
+        self.reg_loss = EmbLoss()
+        self.train_graph = self.csr2tensor(self.create_adjust_matrix(is_sub=False))
+        self.restore_user_e = None
+        self.restore_item_e = None
+        self.apply(xavier_uniform_initialization)
+        self.other_parameter_name = ['restore_user_e', 'restore_item_e']
+
+    def graph_construction(self):
+        r"""Devise three operators to generate the views — node dropout, edge dropout, and random walk of a node.
+
+        """
+        self.sub_graph1 = []
+        if self.type == "ND" or self.type == "ED":
+            self.sub_graph1 = self.csr2tensor(self.create_adjust_matrix(is_sub=True))
+        elif self.type == "RW":
+            for i in range(self.n_layers):
+                _g = self.csr2tensor(self.create_adjust_matrix(is_sub=True))
+                self.sub_graph1.append(_g)
+
+        self.sub_graph2 = []
+        if self.type == "ND" or self.type == "ED":
+            self.sub_graph2 = self.csr2tensor(self.create_adjust_matrix(is_sub=True))
+        elif self.type == "RW":
+            for i in range(self.n_layers):
+                _g = self.csr2tensor(self.create_adjust_matrix(is_sub=True))
+                self.sub_graph2.append(_g)
+
+    def rand_sample(self, high, size=None, replace=True):
+        r"""Randomly discard some points or edges.
+
+        Args:
+            high (int): Upper limit of index value
+            size (int): Array size after sampling
+
+        Returns:
+            numpy.ndarray: Array index after sampling, shape: [size]
+        """
+
+        a = np.arange(high)
+        sample = np.random.choice(a, size=size, replace=replace)
+        return sample
+
+    def create_adjust_matrix(self, is_sub: bool):
+        r"""Get the normalized interaction matrix of users and items.
+
+        Construct the square matrix from the training data and normalize it
+        using the laplace matrix.If it is a subgraph, it may be processed by
+        node dropout or edge dropout.
+
+        .. math::
+            A_{hat} = D^{-0.5} \times A \times D^{-0.5}
+
+        Returns:
+            csr_matrix of the normalized interaction matrix.
+        """
+        matrix = None
+        if not is_sub:
+            ratings = np.ones_like(self._user, dtype=np.float32)
+            matrix = sp.csr_matrix((ratings, (self._user, self._item + self.n_users)),
+                                   shape=(self.n_users + self.n_items, self.n_users + self.n_items))
+        else:
+            if self.type == "ND":
+                drop_user = self.rand_sample(self.n_users, size=int(self.n_users * self.drop_ratio), replace=False)
+                drop_item = self.rand_sample(self.n_items, size=int(self.n_items * self.drop_ratio), replace=False)
+                R_user = np.ones(self.n_users, dtype=np.float32)
+                R_user[drop_user] = 0.
+                R_item = np.ones(self.n_items, dtype=np.float32)
+                R_item[drop_item] = 0.
+                R_user = sp.diags(R_user)
+                R_item = sp.diags(R_item)
+                R_G = sp.csr_matrix((np.ones_like(self._user, dtype=np.float32), (self._user, self._item)),
+                                    shape=(self.n_users, self.n_items))
+                res = R_user.dot(R_G)
+                res = res.dot(R_item)
+
+                user, item = res.nonzero()
+                ratings = res.data
+                matrix = sp.csr_matrix((ratings, (user, item + self.n_users)), shape=(self.n_users + self.n_items, self.n_users + self.n_items))
+
+            elif self.type == "ED" or self.type == "RW":
+                keep_item = self.rand_sample(
+                    len(self._user), size=int(len(self._user) * (1 - self.drop_ratio)), replace=False
+                )
+                user = self._user[keep_item]
+                item = self._item[keep_item]
+
+                matrix = sp.csr_matrix((np.ones_like(user), (user, item + self.n_users)),
+                                       shape=(self.n_users + self.n_items, self.n_users + self.n_items))
+
+        matrix = matrix + matrix.T
+        D = np.array(matrix.sum(axis=1)) + 1e-7
+        D = np.power(D, -0.5).flatten()
+        D = sp.diags(D)
+        return D.dot(matrix).dot(D)
+
+    def csr2tensor(self, matrix: sp.csr_matrix):
+        r"""Convert csr_matrix to tensor.
+
+        Args:
+            matrix (scipy.csr_matrix): Sparse matrix to be converted.
+
+        Returns:
+            torch.sparse.FloatTensor: Transformed sparse matrix.
+        """
+        matrix = matrix.tocoo()
+        x = torch.sparse.FloatTensor(
+            torch.LongTensor([matrix.row.tolist(), matrix.col.tolist()]),
+            torch.FloatTensor(matrix.data.astype(np.float32)), matrix.shape
+        ).to(self.device)
+        return x
+
+    def forward(self, graph):
+        main_ego = torch.cat([self.user_embedding.weight, self.item_embedding.weight])
+        all_ego = [main_ego]
+        if isinstance(graph, list):
+            for sub_graph in graph:
+                main_ego = torch.sparse.mm(sub_graph, main_ego)
+                all_ego.append(main_ego)
+        else:
+            for i in range(self.n_layers):
+                main_ego = torch.sparse.mm(graph, main_ego)
+                all_ego.append(main_ego)
+        all_ego = torch.stack(all_ego, dim=1)
+        all_ego = torch.mean(all_ego, dim=1, keepdim=False)
+        user_emd, item_emd = torch.split(all_ego, [self.n_users, self.n_items], dim=0)
+
+        return user_emd, item_emd
+
+    def calculate_loss(self, interaction):
+        if self.restore_user_e is not None or self.restore_item_e is not None:
+            self.restore_user_e, self.restore_item_e = None, None
+        
+        user_list = interaction[self.USER_ID]
+        pos_item_list = interaction[self.ITEM_ID]
+        neg_item_list = interaction[self.NEG_ITEM_ID]
+        user_emd, item_emd = self.forward(self.train_graph)
+        user_sub1, item_sub1 = self.forward(self.sub_graph1)
+        user_sub2, item_sub2 = self.forward(self.sub_graph2)
+        total_loss = self.calc_bpr_loss(user_emd,item_emd,user_list,pos_item_list,neg_item_list) + \
+            self.calc_ssl_loss(user_list,pos_item_list,user_sub1,user_sub2,item_sub1,item_sub2)
+        return total_loss
+
+    def calc_bpr_loss(self, user_emd, item_emd, user_list, pos_item_list, neg_item_list):
+        r"""Calculate the the pairwise Bayesian Personalized Ranking (BPR) loss and parameter regularization loss.
+
+        Args:
+            user_emd (torch.Tensor): Ego embedding of all users after forwarding.
+            item_emd (torch.Tensor): Ego embedding of all items after forwarding.
+            user_list (torch.Tensor): List of the user.
+            pos_item_list (torch.Tensor): List of positive examples.
+            neg_item_list (torch.Tensor): List of negative examples.
+
+        Returns:
+            torch.Tensor: Loss of BPR tasks and parameter regularization.
+        """
+        u_e = user_emd[user_list]
+        pi_e = item_emd[pos_item_list]
+        ni_e = item_emd[neg_item_list]
+        p_scores = torch.mul(u_e, pi_e).sum(dim=1)
+        n_scores = torch.mul(u_e, ni_e).sum(dim=1)
+
+        l1 = torch.sum(-F.logsigmoid(p_scores - n_scores))
+
+        u_e_p = self.user_embedding(user_list)
+        pi_e_p = self.item_embedding(pos_item_list)
+        ni_e_p = self.item_embedding(neg_item_list)
+
+        l2 = self.reg_loss(u_e_p, pi_e_p, ni_e_p)
+
+        return l1 + l2 * self.reg_weight
+
+    def calc_ssl_loss(self, user_list, pos_item_list, user_sub1, user_sub2, item_sub1, item_sub2):
+        r"""Calculate the loss of self-supervised tasks.
+
+        Args:
+            user_list (torch.Tensor): List of the user.
+            pos_item_list (torch.Tensor): List of positive examples.
+            user_sub1 (torch.Tensor): Ego embedding of all users in the first subgraph after forwarding.
+            user_sub2 (torch.Tensor): Ego embedding of all users in the second subgraph after forwarding.
+            item_sub1 (torch.Tensor): Ego embedding of all items in the first subgraph after forwarding.
+            item_sub2 (torch.Tensor): Ego embedding of all items in the second subgraph after forwarding.
+
+        Returns:
+            torch.Tensor: Loss of self-supervised tasks.
+        """
+
+        u_emd1 = F.normalize(user_sub1[user_list], dim=1)
+        u_emd2 = F.normalize(user_sub2[user_list], dim=1)
+        all_user2 = F.normalize(user_sub2,dim=1)
+        v1 = torch.sum(u_emd1 * u_emd2, dim=1)
+        v2 = u_emd1.matmul(all_user2.T)
+        v1 = torch.exp(v1 / self.ssl_tau)
+        v2 = torch.sum(torch.exp(v2 / self.ssl_tau), dim=1)
+        ssl_user = -torch.sum(torch.log(v1 / v2))
+
+        i_emd1 = F.normalize(item_sub1[pos_item_list], dim=1)
+        i_emd2 = F.normalize(item_sub2[pos_item_list], dim=1)
+        all_item2 = F.normalize(item_sub2,dim=1)
+        v3 = torch.sum(i_emd1 * i_emd2, dim=1)
+        v4 = i_emd1.matmul(all_item2.T)
+        v3 = torch.exp(v3 / self.ssl_tau)
+        v4 = torch.sum(torch.exp(v4 / self.ssl_tau), dim=1)
+        ssl_item = -torch.sum(torch.log(v3 / v4))
+
+        return (ssl_item + ssl_user) * self.ssl_weight
+
+    def predict(self, interaction):
+        if self.restore_user_e is None or self.restore_item_e is None:
+            self.restore_user_e, self.restore_item_e = self.forward(self.train_graph)
+
+        user = self.restore_user_e[interaction[self.USER_ID]]
+        item = self.restore_item_e[interaction[self.ITEM_ID]]
+        return torch.sum(user * item, dim=1)
+
+    def full_sort_predict(self, interaction):
+        if self.restore_user_e is None or self.restore_item_e is None:
+            self.restore_user_e, self.restore_item_e = self.forward(self.train_graph)
+        
+        user = self.restore_user_e[interaction[self.USER_ID]]
+        return user.matmul(self.restore_item_e.T)
+
+    def train(self, mode: bool = True):
+        r"""Override train method of base class.The subgraph is reconstructed each time it is called.
+
+        """
+        T = super().train(mode=mode)
+        if mode:
+            self.graph_construction()
+        return T
diff --git a/recbole/model/layers.py b/recbole/model/layers.py
index b2a0f3b0d..ea04deecb 100644
--- a/recbole/model/layers.py
+++ b/recbole/model/layers.py
@@ -5,9 +5,9 @@
 # @File   : layers.py
 
 # UPDATE:
-# @Time   : 2020/8/24 14:58, 2020/9/16, 2020/9/21, 2020/10/9
-# @Author : Yujie Lu, Xingyu Pan, Zhichao Feng, Hui Wang
-# @Email  : yujielu1998@gmail.com, panxy@ruc.edu.cn, fzcbupt@gmail.com, hui.wang@ruc.edu.cn
+# @Time   : 2020/8/24 14:58, 2020/9/16, 2020/9/21, 2020/10/9, 2021/05/01
+# @Author : Yujie Lu, Xingyu Pan, Zhichao Feng, Hui Wang, Xinyan Fan
+# @Email  : yujielu1998@gmail.com, panxy@ruc.edu.cn, fzcbupt@gmail.com, hui.wang@ruc.edu.cn, xinyan.fan@ruc.edu.cn
 
 """
 recbole.model.layers
@@ -359,11 +359,13 @@ def __init__(self, n_heads, hidden_size, hidden_dropout_prob, attn_dropout_prob,
         self.num_attention_heads = n_heads
         self.attention_head_size = int(hidden_size / n_heads)
         self.all_head_size = self.num_attention_heads * self.attention_head_size
+        self.sqrt_attention_head_size = math.sqrt(self.attention_head_size)
 
         self.query = nn.Linear(hidden_size, self.all_head_size)
         self.key = nn.Linear(hidden_size, self.all_head_size)
         self.value = nn.Linear(hidden_size, self.all_head_size)
 
+        self.softmax = nn.Softmax(dim=-1)
         self.attn_dropout = nn.Dropout(attn_dropout_prob)
 
         self.dense = nn.Linear(hidden_size, hidden_size)
@@ -373,28 +375,28 @@ def __init__(self, n_heads, hidden_size, hidden_dropout_prob, attn_dropout_prob,
     def transpose_for_scores(self, x):
         new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
         x = x.view(*new_x_shape)
-        return x.permute(0, 2, 1, 3)
+        return x
 
     def forward(self, input_tensor, attention_mask):
         mixed_query_layer = self.query(input_tensor)
         mixed_key_layer = self.key(input_tensor)
         mixed_value_layer = self.value(input_tensor)
 
-        query_layer = self.transpose_for_scores(mixed_query_layer)
-        key_layer = self.transpose_for_scores(mixed_key_layer)
-        value_layer = self.transpose_for_scores(mixed_value_layer)
+        query_layer = self.transpose_for_scores(mixed_query_layer).permute(0, 2, 1, 3)
+        key_layer = self.transpose_for_scores(mixed_key_layer).permute(0, 2, 3, 1)
+        value_layer = self.transpose_for_scores(mixed_value_layer).permute(0, 2, 1, 3)
 
         # Take the dot product between "query" and "key" to get the raw attention scores.
-        attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
+        attention_scores = torch.matmul(query_layer, key_layer)
 
-        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
+        attention_scores = attention_scores / self.sqrt_attention_head_size
         # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
         # [batch_size heads seq_len seq_len] scores
         # [batch_size 1 1 seq_len]
         attention_scores = attention_scores + attention_mask
 
         # Normalize the attention scores to probabilities.
-        attention_probs = nn.Softmax(dim=-1)(attention_scores)
+        attention_probs = self.softmax(attention_scores)
         # This is actually dropping out entire tokens to attend to, which might
         # seem a bit unusual, but is taken from the original Transformer paper.
 
@@ -551,6 +553,174 @@ def forward(self, hidden_states, attention_mask, output_all_encoded_layers=True)
         return all_encoder_layers
 
 
+class ItemToInterestAggregation(nn.Module):
+    def __init__(self, seq_len, hidden_size, k_interests=5):
+        super().__init__()
+        self.k_interests = k_interests # k latent interests
+        self.theta = nn.Parameter(torch.randn([hidden_size, k_interests]))
+        
+    def forward(self, input_tensor): # [B, L, d] -> [B, k, d]
+        D_matrix = torch.matmul(input_tensor, self.theta) #[B, L, k]
+        D_matrix = nn.Softmax(dim=-2)(D_matrix)
+        result = torch.einsum('nij, nik -> nkj', input_tensor, D_matrix) # #[B, k, d]
+
+        return result
+
+
+class LightMultiHeadAttention(nn.Module):
+    def __init__(self, n_heads, k_interests, hidden_size, seq_len, hidden_dropout_prob, attn_dropout_prob, layer_norm_eps):
+        super(LightMultiHeadAttention, self).__init__()
+        if hidden_size % n_heads != 0:
+            raise ValueError(
+                "The hidden size (%d) is not a multiple of the number of attention "
+                "heads (%d)" % (hidden_size, n_heads))
+
+        self.num_attention_heads = n_heads
+        self.attention_head_size = int(hidden_size / n_heads)
+        self.all_head_size = self.num_attention_heads * self.attention_head_size  
+
+        # initialization for low-rank decomposed self-attention
+        self.query = nn.Linear(hidden_size, self.all_head_size)
+        self.key = nn.Linear(hidden_size, self.all_head_size)
+        self.value = nn.Linear(hidden_size, self.all_head_size)
+
+        self.attpooling_key = ItemToInterestAggregation(seq_len, hidden_size, k_interests)
+        self.attpooling_value = ItemToInterestAggregation(seq_len, hidden_size, k_interests)
+
+        # initialization for decoupled position encoding
+        self.attn_scale_factor = 2
+        self.pos_q_linear = nn.Linear(hidden_size, self.all_head_size)
+        self.pos_k_linear = nn.Linear(hidden_size, self.all_head_size)
+        self.pos_scaling = float(self.attention_head_size * self.attn_scale_factor) ** -0.5
+        self.pos_ln = nn.LayerNorm(hidden_size, eps=layer_norm_eps)
+
+        self.attn_dropout = nn.Dropout(attn_dropout_prob)
+
+        self.dense = nn.Linear(hidden_size, hidden_size)
+        self.LayerNorm = nn.LayerNorm(hidden_size, eps=layer_norm_eps)
+        self.out_dropout = nn.Dropout(hidden_dropout_prob)
+
+    def transpose_for_scores(self, x): # transfor to multihead
+        new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
+        x = x.view(*new_x_shape)
+        return x.permute(0, 2, 1, 3)
+
+    def forward(self, input_tensor, pos_emb):
+        # linear map
+        mixed_query_layer = self.query(input_tensor)
+        mixed_key_layer = self.key(input_tensor)
+        mixed_value_layer = self.value(input_tensor)
+
+        # low-rank decomposed self-attention: relation of items
+        query_layer = self.transpose_for_scores(mixed_query_layer)
+        key_layer = self.transpose_for_scores(self.attpooling_key(mixed_key_layer))
+        value_layer = self.transpose_for_scores(self.attpooling_value(mixed_value_layer))
+
+        attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
+        attention_scores = attention_scores / math.sqrt(self.attention_head_size)
+        
+        # normalize the attention scores to probabilities.
+        attention_probs = nn.Softmax(dim=-2)(attention_scores)
+        attention_probs = self.attn_dropout(attention_probs)
+        context_layer_item = torch.matmul(attention_probs, value_layer)
+
+        # decoupled position encoding: relation of positions
+        value_layer_pos = self.transpose_for_scores(mixed_value_layer)
+        pos_emb = self.pos_ln(pos_emb).unsqueeze(0)
+        pos_query_layer = self.transpose_for_scores(self.pos_q_linear(pos_emb)) * self.pos_scaling
+        pos_key_layer = self.transpose_for_scores(self.pos_k_linear(pos_emb))
+
+        abs_pos_bias = torch.matmul(pos_query_layer, pos_key_layer.transpose(-1, -2))
+        abs_pos_bias = abs_pos_bias / math.sqrt(self.attention_head_size)
+        abs_pos_bias = nn.Softmax(dim=-2)(abs_pos_bias)
+
+        context_layer_pos = torch.matmul(abs_pos_bias, value_layer_pos)
+
+        context_layer = context_layer_item + context_layer_pos
+
+        context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
+        new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
+        context_layer = context_layer.view(*new_context_layer_shape)
+        hidden_states = self.dense(context_layer)
+        hidden_states = self.out_dropout(hidden_states)
+        hidden_states = self.LayerNorm(hidden_states + input_tensor)
+
+        return hidden_states
+
+
+class LightTransformerLayer(nn.Module):
+    """
+    One transformer layer consists of a multi-head self-attention layer and a point-wise feed-forward layer.
+    Args:
+        hidden_states (torch.Tensor): the input of the multi-head self-attention sublayer
+        attention_mask (torch.Tensor): the attention mask for the multi-head self-attention sublayer
+    Returns:
+        feedforward_output (torch.Tensor): the output of the point-wise feed-forward sublayer, is the output of the transformer layer
+    """
+    def __init__(self, n_heads, k_interests, hidden_size, seq_len, intermediate_size,
+                 hidden_dropout_prob, attn_dropout_prob, hidden_act, layer_norm_eps):
+        super(LightTransformerLayer, self).__init__()
+        self.multi_head_attention = LightMultiHeadAttention(n_heads, k_interests, hidden_size,
+                                       seq_len, hidden_dropout_prob, attn_dropout_prob, layer_norm_eps)
+        self.feed_forward = FeedForward(hidden_size, intermediate_size,
+                                         hidden_dropout_prob, hidden_act, layer_norm_eps)
+
+    def forward(self, hidden_states, pos_emb):
+        attention_output = self.multi_head_attention(hidden_states, pos_emb)
+        feedforward_output = self.feed_forward(attention_output)
+        return feedforward_output
+
+
+class LightTransformerEncoder(nn.Module):
+    r""" One LightTransformerEncoder consists of several LightTransformerLayers.
+        - n_layers(num): num of transformer layers in transformer encoder. Default: 2
+        - n_heads(num): num of attention heads for multi-head attention layer. Default: 2
+        - hidden_size(num): the input and output hidden size. Default: 64
+        - inner_size(num): the dimensionality in feed-forward layer. Default: 256
+        - hidden_dropout_prob(float): probability of an element to be zeroed. Default: 0.5
+        - attn_dropout_prob(float): probability of an attention score to be zeroed. Default: 0.5
+        - hidden_act(str): activation function in feed-forward layer. Default: 'gelu'
+                      candidates: 'gelu', 'relu', 'swish', 'tanh', 'sigmoid'
+        - layer_norm_eps(float): a value added to the denominator for numerical stability. Default: 1e-12
+    """
+    def __init__(self,
+                 n_layers=2,
+                 n_heads=2,
+                 k_interests=5,
+                 hidden_size=64,
+                 seq_len=50,
+                 inner_size=256,
+                 hidden_dropout_prob=0.5,
+                 attn_dropout_prob=0.5,
+                 hidden_act='gelu',
+                 layer_norm_eps=1e-12):
+
+        super(LightTransformerEncoder, self).__init__()
+        layer = LightTransformerLayer(n_heads, k_interests, hidden_size, seq_len, inner_size,
+                                 hidden_dropout_prob, attn_dropout_prob, hidden_act, layer_norm_eps)
+        self.layer = nn.ModuleList([copy.deepcopy(layer)
+                                    for _ in range(n_layers)])
+
+    def forward(self, hidden_states, pos_emb, output_all_encoded_layers=True):
+        """
+        Args:
+            hidden_states (torch.Tensor): the input of the TrandformerEncoder
+            attention_mask (torch.Tensor): the attention mask for the input hidden_states
+            output_all_encoded_layers (Bool): whether output all transformer layers' output
+        Returns:
+            all_encoder_layers (list): if output_all_encoded_layers is True, return a list consists of all transformer layers' output,
+            otherwise return a list only consists of the output of last transformer layer.
+        """
+        all_encoder_layers = []
+        for layer_module in self.layer:
+            hidden_states = layer_module(hidden_states, pos_emb)
+            if output_all_encoded_layers:
+                all_encoder_layers.append(hidden_states)
+        if not output_all_encoded_layers:
+            all_encoder_layers.append(hidden_states)
+        return all_encoder_layers
+
+
 class ContextSeqEmbAbstractLayer(nn.Module):
     """For Deep Interest Network and feature-rich sequential recommender systems, return features embedding matrices."""
 
diff --git a/recbole/model/sequential_recommender/__init__.py b/recbole/model/sequential_recommender/__init__.py
index 4d1c15dfb..88928f45d 100644
--- a/recbole/model/sequential_recommender/__init__.py
+++ b/recbole/model/sequential_recommender/__init__.py
@@ -9,11 +9,13 @@
 from recbole.model.sequential_recommender.gru4recf import GRU4RecF
 from recbole.model.sequential_recommender.gru4reckg import GRU4RecKG
 from recbole.model.sequential_recommender.ksr import KSR
+from recbole.model.sequential_recommender.lightsans import LightSANs
 from recbole.model.sequential_recommender.narm import NARM
 from recbole.model.sequential_recommender.nextitnet import NextItNet
 from recbole.model.sequential_recommender.s3rec import S3Rec
 from recbole.model.sequential_recommender.sasrec import SASRec
 from recbole.model.sequential_recommender.sasrecf import SASRecF
+from recbole.model.sequential_recommender.sine import SINE
 from recbole.model.sequential_recommender.srgnn import SRGNN
 from recbole.model.sequential_recommender.stamp import STAMP
 from recbole.model.sequential_recommender.transrec import TransRec
diff --git a/recbole/model/sequential_recommender/bert4rec.py b/recbole/model/sequential_recommender/bert4rec.py
index d524a0201..14c18f68b 100644
--- a/recbole/model/sequential_recommender/bert4rec.py
+++ b/recbole/model/sequential_recommender/bert4rec.py
@@ -87,15 +87,6 @@ def _init_weights(self, module):
         if isinstance(module, nn.Linear) and module.bias is not None:
             module.bias.data.zero_()
 
-    def get_attention_mask(self, item_seq):
-        """Generate bidirectional attention mask for multi-head attention."""
-        attention_mask = (item_seq > 0).long()
-        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.int64
-        # bidirectional mask
-        extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
-        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
-        return extended_attention_mask
-
     def _neg_sample(self, item_set):
         item = random.randint(1, self.n_items - 1)
         while item in item_set:
@@ -173,7 +164,7 @@ def forward(self, item_seq):
         input_emb = item_emb + position_embedding
         input_emb = self.LayerNorm(input_emb)
         input_emb = self.dropout(input_emb)
-        extended_attention_mask = self.get_attention_mask(item_seq)
+        extended_attention_mask = self.get_attention_mask(item_seq, bidirectional=True)
         trm_output = self.trm_encoder(input_emb, extended_attention_mask, output_all_encoded_layers=True)
         output = trm_output[-1]
         return output  # [B L H]
diff --git a/recbole/model/sequential_recommender/dien.py b/recbole/model/sequential_recommender/dien.py
index 85c1c5eff..e7f53e066 100644
--- a/recbole/model/sequential_recommender/dien.py
+++ b/recbole/model/sequential_recommender/dien.py
@@ -58,8 +58,14 @@ def __init__(self, config, dataset):
         self.user_feat = dataset.get_user_feature()
         self.item_feat = dataset.get_item_feature()
 
-        num_item_feature = len(self.item_feat.interaction)
-        num_user_feature = len(self.user_feat.interaction)
+        num_item_feature = sum(
+            1 if dataset.field2type[field] != FeatureType.FLOAT_SEQ else dataset.num(field)
+            for field in self.item_feat.interaction.keys()
+        )
+        num_user_feature = sum(
+            1 if dataset.field2type[field] != FeatureType.FLOAT_SEQ else dataset.num(field)
+            for field in self.user_feat.interaction.keys()
+        )
         item_feat_dim = num_item_feature * self.embedding_size
         mask_mat = torch.arange(self.max_seq_length).to(self.device).view(1, -1)  # init mask
 
diff --git a/recbole/model/sequential_recommender/din.py b/recbole/model/sequential_recommender/din.py
index 709f0e2f0..101a2b4d4 100644
--- a/recbole/model/sequential_recommender/din.py
+++ b/recbole/model/sequential_recommender/din.py
@@ -26,7 +26,7 @@
 
 from recbole.model.abstract_recommender import SequentialRecommender
 from recbole.model.layers import MLPLayers, SequenceAttLayer, ContextSeqEmbLayer
-from recbole.utils import InputType
+from recbole.utils import InputType, FeatureType
 
 
 class DIN(SequentialRecommender):
@@ -60,7 +60,10 @@ def __init__(self, config, dataset):
         # init MLP layers
         # self.dnn_list = [(3 * self.num_feature_field['item'] + self.num_feature_field['user'])
         #                  * self.embedding_size] + self.mlp_hidden_size
-        num_item_feature = len(self.item_feat.interaction.keys())
+        num_item_feature = sum(
+            1 if dataset.field2type[field] != FeatureType.FLOAT_SEQ else dataset.num(field)
+            for field in self.item_feat.interaction.keys()
+        )
         self.dnn_list = [3 * num_item_feature * self.embedding_size] + self.mlp_hidden_size
         self.att_list = [4 * num_item_feature * self.embedding_size] + self.mlp_hidden_size
 
diff --git a/recbole/model/sequential_recommender/fdsa.py b/recbole/model/sequential_recommender/fdsa.py
index f0a193b52..ccd4823b9 100644
--- a/recbole/model/sequential_recommender/fdsa.py
+++ b/recbole/model/sequential_recommender/fdsa.py
@@ -107,22 +107,6 @@ def _init_weights(self, module):
         if isinstance(module, nn.Linear) and module.bias is not None:
             module.bias.data.zero_()
 
-    def get_attention_mask(self, item_seq):
-        """Generate left-to-right uni-directional attention mask for multi-head attention."""
-        attention_mask = (item_seq > 0).long()
-        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.int64
-        # mask for left-to-right unidirectional
-        max_len = attention_mask.size(-1)
-        attn_shape = (1, max_len, max_len)
-        subsequent_mask = torch.triu(torch.ones(attn_shape), diagonal=1)  # torch.uint8
-        subsequent_mask = (subsequent_mask == 0).unsqueeze(1)
-        subsequent_mask = subsequent_mask.long().to(item_seq.device)
-
-        extended_attention_mask = extended_attention_mask * subsequent_mask
-        extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
-        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
-        return extended_attention_mask
-
     def forward(self, item_seq, item_seq_len):
         item_emb = self.item_embedding(item_seq)
 
diff --git a/recbole/model/sequential_recommender/gcsan.py b/recbole/model/sequential_recommender/gcsan.py
index e64381473..18a9fdcda 100644
--- a/recbole/model/sequential_recommender/gcsan.py
+++ b/recbole/model/sequential_recommender/gcsan.py
@@ -156,22 +156,6 @@ def _init_weights(self, module):
         if isinstance(module, nn.Linear) and module.bias is not None:
             module.bias.data.zero_()
 
-    def get_attention_mask(self, item_seq):
-        """Generate left-to-right uni-directional attention mask for multi-head attention."""
-        attention_mask = (item_seq > 0).long()
-        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.int64
-        # mask for left-to-right unidirectional
-        max_len = attention_mask.size(-1)
-        attn_shape = (1, max_len, max_len)
-        subsequent_mask = torch.triu(torch.ones(attn_shape), diagonal=1)  # torch.uint8
-        subsequent_mask = (subsequent_mask == 0).unsqueeze(1)
-        subsequent_mask = subsequent_mask.long().to(item_seq.device)
-
-        extended_attention_mask = extended_attention_mask * subsequent_mask
-        extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
-        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
-        return extended_attention_mask
-
     def _get_slice(self, item_seq):
         items, n_node, A, alias_inputs = [], [], [], []
         max_n_node = item_seq.size(1)
diff --git a/recbole/model/sequential_recommender/hrm.py b/recbole/model/sequential_recommender/hrm.py
index 421835266..3dd1b0ae1 100644
--- a/recbole/model/sequential_recommender/hrm.py
+++ b/recbole/model/sequential_recommender/hrm.py
@@ -156,7 +156,6 @@ def predict(self, interaction):
         test_item = interaction[self.ITEM_ID]
         user = interaction[self.USER_ID]
         seq_output = self.forward(item_seq, user, seq_item_len)
-        seq_output = seq_output.repeat(1, self.embedding_size)
         test_item_emb = self.item_embedding(test_item)
         scores = torch.mul(seq_output, test_item_emb).sum(dim=1)
 
diff --git a/recbole/model/sequential_recommender/lightsans.py b/recbole/model/sequential_recommender/lightsans.py
new file mode 100644
index 000000000..5119166bf
--- /dev/null
+++ b/recbole/model/sequential_recommender/lightsans.py
@@ -0,0 +1,129 @@
+# -*- coding: utf-8 -*-
+# @Time    : 2021/05/01
+# @Author  : Xinyan Fan
+# @Email   : xinyan.fan@ruc.edu.cn
+
+"""
+LightSANs
+################################################
+Reference:
+    Xin-Yan Fan et al. "Lighter and Better: Low-Rank Decomposed Self-Attention Networks for Next-Item Recommendation." in SIGIR 2021.
+Reference:
+    https://github.com/BELIEVEfxy/LightSANs
+"""
+
+import torch
+from torch import nn
+
+from recbole.model.abstract_recommender import SequentialRecommender
+from recbole.model.loss import BPRLoss
+from recbole.model.layers import LightTransformerEncoder
+
+class LightSANs(SequentialRecommender):
+
+    def __init__(self, config, dataset):
+        super(LightSANs, self).__init__(config, dataset)
+
+        # load parameters info
+        self.n_layers = config['n_layers']
+        self.n_heads = config['n_heads']
+        self.k_interests = config['k_interests']
+        self.hidden_size = config['hidden_size'] # same as embedding_size
+        self.inner_size = config['inner_size']  # the dimensionality in feed-forward layer
+        self.hidden_dropout_prob = config['hidden_dropout_prob']
+        self.attn_dropout_prob = config['attn_dropout_prob']
+        self.hidden_act = config['hidden_act']
+        self.layer_norm_eps = config['layer_norm_eps']
+
+        self.initializer_range = config['initializer_range']
+        self.loss_type = config['loss_type']
+
+        self.seq_len = self.max_seq_length
+        # define layers and loss
+        self.item_embedding = nn.Embedding(self.n_items, self.hidden_size , padding_idx=0)
+        self.position_embedding = nn.Embedding(self.max_seq_length, self.hidden_size)
+        self.trm_encoder = LightTransformerEncoder(n_layers=self.n_layers, n_heads=self.n_heads, 
+                                              k_interests=self.k_interests, hidden_size=self.hidden_size, 
+                                              seq_len=self.seq_len,
+                                              inner_size=self.inner_size,
+                                              hidden_dropout_prob=self.hidden_dropout_prob,
+                                              attn_dropout_prob=self.attn_dropout_prob,
+                                              hidden_act=self.hidden_act, layer_norm_eps=self.layer_norm_eps)
+
+        self.LayerNorm = nn.LayerNorm(self.hidden_size, eps=self.layer_norm_eps)
+        self.dropout = nn.Dropout(self.hidden_dropout_prob)
+
+        if self.loss_type == 'BPR':
+            self.loss_fct = BPRLoss()
+        elif self.loss_type == 'CE':
+            self.loss_fct = nn.CrossEntropyLoss()
+        else:
+            raise NotImplementedError("Make sure 'loss_type' in ['BPR', 'CE']!")
+
+        # parameters initialization
+        self.apply(self._init_weights)
+
+    def _init_weights(self, module):
+        """ Initialize the weights """
+        if isinstance(module, (nn.Linear, nn.Embedding)):
+            module.weight.data.normal_(mean=0.0, std=self.initializer_range)
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+        if isinstance(module, nn.Linear) and module.bias is not None:
+            module.bias.data.zero_()
+    
+    def embedding_layer(self, item_seq):
+        position_ids = torch.arange(item_seq.size(1), dtype=torch.long, device=item_seq.device)
+        position_embedding = self.position_embedding(position_ids)
+        item_emb = self.item_embedding(item_seq)
+        return item_emb, position_embedding
+
+    def forward(self, item_seq, item_seq_len):
+        item_emb, position_embedding = self.embedding_layer(item_seq)
+        item_emb = self.LayerNorm(item_emb)
+        item_emb = self.dropout(item_emb)
+
+        trm_output = self.trm_encoder(item_emb,
+                                      position_embedding,
+                                      output_all_encoded_layers=True)
+        output = trm_output[-1]
+        output = self.gather_indexes(output, item_seq_len - 1)
+        return output # [B H]
+
+    def calculate_loss(self, interaction):
+        item_seq = interaction[self.ITEM_SEQ]
+        item_seq_len = interaction[self.ITEM_SEQ_LEN]
+        seq_output = self.forward(item_seq, item_seq_len)
+        pos_items = interaction[self.POS_ITEM_ID]
+        if self.loss_type == 'BPR':
+            neg_items = interaction[self.NEG_ITEM_ID]
+            pos_items_emb = self.item_embedding(pos_items)
+            neg_items_emb = self.item_embedding(neg_items)
+            pos_score = torch.sum(seq_output * pos_items_emb, dim=-1)  # [B]
+            neg_score = torch.sum(seq_output * neg_items_emb, dim=-1)  # [B]
+            loss = self.loss_fct(pos_score, neg_score)
+            return loss
+        else:  # self.loss_type = 'CE'
+            test_item_emb = self.item_embedding.weight
+            logits = torch.matmul(seq_output, test_item_emb.transpose(0, 1))
+            loss = self.loss_fct(logits, pos_items)
+            return loss
+
+    def predict(self, interaction):
+        item_seq = interaction[self.ITEM_SEQ]
+        item_seq_len = interaction[self.ITEM_SEQ_LEN]
+        test_item = interaction[self.ITEM_ID]
+        
+        seq_output = self.forward(item_seq, item_seq_len)
+        test_item_emb = self.item_embedding(test_item)
+        scores = torch.mul(seq_output, test_item_emb).sum(dim=1)  # [B]
+        return scores
+
+    def full_sort_predict(self, interaction):
+        item_seq = interaction[self.ITEM_SEQ]
+        item_seq_len = interaction[self.ITEM_SEQ_LEN]
+        seq_output = self.forward(item_seq, item_seq_len)
+        test_items_emb = self.item_embedding.weight
+        scores = torch.matmul(seq_output, test_items_emb.transpose(0, 1))  # [B n_items]
+        return scores
\ No newline at end of file
diff --git a/recbole/model/sequential_recommender/s3rec.py b/recbole/model/sequential_recommender/s3rec.py
index 732bab61d..d61cccca5 100644
--- a/recbole/model/sequential_recommender/s3rec.py
+++ b/recbole/model/sequential_recommender/s3rec.py
@@ -152,25 +152,6 @@ def _segment_prediction(self, context, segment_emb):
         score = torch.mul(context, segment_emb)  # [B H]
         return torch.sigmoid(torch.sum(score, dim=-1))  # [B]
 
-    def get_attention_mask(self, sequence, bidirectional=True):
-        """
-        In the pre-training stage, we generate bidirectional attention mask for multi-head attention.
-
-        In the fine-tuning stage, we generate left-to-right uni-directional attention mask for multi-head attention.
-        """
-        attention_mask = (sequence > 0).long()
-        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.int64
-        if not bidirectional:
-            max_len = attention_mask.size(-1)
-            attn_shape = (1, max_len, max_len)
-            subsequent_mask = torch.triu(torch.ones(attn_shape), diagonal=1)  # torch.uint8
-            subsequent_mask = (subsequent_mask == 0).unsqueeze(1)
-            subsequent_mask = subsequent_mask.long().to(sequence.device)
-            extended_attention_mask = extended_attention_mask * subsequent_mask
-        extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
-        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
-        return extended_attention_mask
-
     def forward(self, item_seq, bidirectional=True):
         position_ids = torch.arange(item_seq.size(1), dtype=torch.long, device=item_seq.device)
         position_ids = position_ids.unsqueeze(0).expand_as(item_seq)
diff --git a/recbole/model/sequential_recommender/sasrec.py b/recbole/model/sequential_recommender/sasrec.py
index ea58a8fdf..ca16028fc 100644
--- a/recbole/model/sequential_recommender/sasrec.py
+++ b/recbole/model/sequential_recommender/sasrec.py
@@ -88,22 +88,6 @@ def _init_weights(self, module):
         if isinstance(module, nn.Linear) and module.bias is not None:
             module.bias.data.zero_()
 
-    def get_attention_mask(self, item_seq):
-        """Generate left-to-right uni-directional attention mask for multi-head attention."""
-        attention_mask = (item_seq > 0).long()
-        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.int64
-        # mask for left-to-right unidirectional
-        max_len = attention_mask.size(-1)
-        attn_shape = (1, max_len, max_len)
-        subsequent_mask = torch.triu(torch.ones(attn_shape), diagonal=1)  # torch.uint8
-        subsequent_mask = (subsequent_mask == 0).unsqueeze(1)
-        subsequent_mask = subsequent_mask.long().to(item_seq.device)
-
-        extended_attention_mask = extended_attention_mask * subsequent_mask
-        extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
-        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
-        return extended_attention_mask
-
     def forward(self, item_seq, item_seq_len):
         position_ids = torch.arange(item_seq.size(1), dtype=torch.long, device=item_seq.device)
         position_ids = position_ids.unsqueeze(0).expand_as(item_seq)
diff --git a/recbole/model/sequential_recommender/sasrecf.py b/recbole/model/sequential_recommender/sasrecf.py
index 1125381d0..497f9fe45 100644
--- a/recbole/model/sequential_recommender/sasrecf.py
+++ b/recbole/model/sequential_recommender/sasrecf.py
@@ -14,6 +14,7 @@
 from recbole.model.abstract_recommender import SequentialRecommender
 from recbole.model.layers import TransformerEncoder, FeatureSeqEmbLayer
 from recbole.model.loss import BPRLoss
+from recbole.utils import FeatureType
 
 
 class SASRecF(SequentialRecommender):
@@ -37,7 +38,10 @@ def __init__(self, config, dataset):
         self.selected_features = config['selected_features']
         self.pooling_mode = config['pooling_mode']
         self.device = config['device']
-        self.num_feature_field = len(config['selected_features'])
+        self.num_feature_field = sum(
+            1 if dataset.field2type[field] != FeatureType.FLOAT_SEQ else dataset.num(field)
+            for field in config['selected_features']
+        )
 
         self.initializer_range = config['initializer_range']
         self.loss_type = config['loss_type']
@@ -88,21 +92,6 @@ def _init_weights(self, module):
         if isinstance(module, nn.Linear) and module.bias is not None:
             module.bias.data.zero_()
 
-    def get_attention_mask(self, item_seq):
-        """Generate left-to-right uni-directional attention mask for multi-head attention."""
-        attention_mask = (item_seq > 0).long()
-        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)  # torch.int64
-        # mask for left-to-right unidirectional
-        max_len = attention_mask.size(-1)
-        attn_shape = (1, max_len, max_len)
-        subsequent_mask = torch.triu(torch.ones(attn_shape), diagonal=1)  # torch.uint8
-        subsequent_mask = (subsequent_mask == 0).unsqueeze(1)
-        subsequent_mask = subsequent_mask.long().to(item_seq.device)
-        extended_attention_mask = extended_attention_mask * subsequent_mask
-        extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
-        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
-        return extended_attention_mask
-
     def forward(self, item_seq, item_seq_len):
         item_emb = self.item_embedding(item_seq)
 
diff --git a/recbole/model/sequential_recommender/sine.py b/recbole/model/sequential_recommender/sine.py
new file mode 100644
index 000000000..d1a052610
--- /dev/null
+++ b/recbole/model/sequential_recommender/sine.py
@@ -0,0 +1,185 @@
+# -*- coding: utf-8 -*-
+# @Time    : 2021/11/23 11:10
+# @Author  : Jingqi Gao
+# @Email   : jgaoaz@connect.ust.hk
+
+r"""
+SINE
+################################################
+
+Reference:
+    Qiaoyu Tan et al. "Sparse-Interest Network for Sequential Recommendation." in WSDM 2021.
+
+"""
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.init import xavier_normal_
+
+from recbole.model.abstract_recommender import SequentialRecommender
+from recbole.model.loss import BPRLoss
+from recbole.utils import InputType
+
+torch.autograd.set_detect_anomaly(True)
+
+
+class SINE(SequentialRecommender):
+    input_type = InputType.PAIRWISE
+
+    def __init__(self, config, dataset):
+        super(SINE, self).__init__(config, dataset)
+
+        # load dataset info
+        self.n_users = dataset.user_num
+        self.n_items = dataset.item_num
+
+        # load parameters info
+        self.device = config["device"]
+        self.embedding_size = config['embedding_size']
+        self.loss_type = config['loss_type']
+        self.layer_norm_eps = config['layer_norm_eps']
+
+        if self.loss_type == 'BPR':
+            self.loss_fct = BPRLoss()
+        elif self.loss_type == 'CE':
+            self.loss_fct = nn.CrossEntropyLoss()
+        elif self.loss_type == 'NLL':
+            self.loss_fct = nn.NLLLoss()
+        else:
+            raise NotImplementedError("Make sure 'loss_type' in ['BPR', 'CE', 'NLL']!")
+
+        self.D = config['embedding_size']
+        self.L = config['prototype_size']  # 500 for movie-len dataset
+        self.k = config['interest_size']  # 4 for movie-len dataset
+        self.tau = config['tau_ratio']  # 0.1 in paper
+        self.reg_loss_ratio = config['reg_loss_ratio']  # 0.1 in paper
+
+        self.initializer_range = 0.01
+
+        self.w1 = self._init_weight((self.D, self.D))
+        self.w2 = self._init_weight(self.D)
+        self.w3 = self._init_weight((self.D, self.D))
+        self.w4 = self._init_weight(self.D)
+
+        self.C = nn.Embedding(self.L, self.D)
+
+        self.w_k_1 = self._init_weight((self.k, self.D, self.D))
+        self.w_k_2 = self._init_weight((self.k, self.D))
+        self.item_embedding = nn.Embedding(self.n_items, self.D, padding_idx=0)
+        self.ln2 = nn.LayerNorm(self.embedding_size, eps=self.layer_norm_eps)
+        self.ln4 = nn.LayerNorm(self.embedding_size, eps=self.layer_norm_eps)
+
+        # parameters initialization
+        self.apply(self._init_weights)
+
+    def _init_weight(self, shape):
+        mat = np.random.normal(0, self.initializer_range, shape)
+        return torch.tensor(mat, dtype=torch.float32, requires_grad=True).to(self.device)
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Embedding):
+            xavier_normal_(module.weight)
+        elif isinstance(module, nn.LayerNorm):
+            module.bias.data.zero_()
+            module.weight.data.fill_(1.0)
+
+    def calculate_loss(self, interaction):
+        item_seq = interaction[self.ITEM_SEQ]
+        item_seq_len = interaction[self.ITEM_SEQ_LEN]
+        seq_output = self.forward(item_seq, item_seq_len)
+        pos_items = interaction[self.POS_ITEM_ID]
+
+        if self.loss_type == 'BPR':
+            neg_items = interaction[self.NEG_ITEM_ID]
+            pos_items_emb = self.item_embedding(pos_items)
+            neg_items_emb = self.item_embedding(neg_items)
+            pos_score = torch.sum(seq_output * pos_items_emb, dim=-1)  # [B]
+            neg_score = torch.sum(seq_output * neg_items_emb, dim=-1)  # [B]
+            loss = self.loss_fct(pos_score, neg_score)
+            return loss
+        elif self.loss_type == 'CE':
+            test_item_emb = self.item_embedding.weight
+            logits = torch.matmul(seq_output, test_item_emb.transpose(0, 1))
+            loss = self.loss_fct(logits, pos_items)
+            return loss
+        else:
+            test_item_emb = self.item_embedding.weight
+            logits = torch.matmul(seq_output, test_item_emb.transpose(0, 1))
+            logits = F.log_softmax(logits, dim=1)
+            loss = self.loss_fct(logits, pos_items)
+            return loss + self.calculate_reg_loss() * self.reg_loss_ratio
+
+    def calculate_reg_loss(self):
+        C_mean = torch.mean(self.C.weight, dim=1, keepdim=True)
+        C_reg = (self.C.weight - C_mean)
+        C_reg = C_reg.matmul(C_reg.T) / self.D
+        return (torch.norm(C_reg) ** 2 - torch.norm(torch.diag(C_reg)) ** 2) / 2
+
+    def predict(self, interaction):
+        item_seq = interaction[self.ITEM_SEQ]
+        item_seq_len = interaction[self.ITEM_SEQ_LEN]
+        test_item = interaction[self.ITEM_ID]
+        seq_output = self.forward(item_seq, item_seq_len)
+        test_item_emb = self.item_embedding(test_item)
+        scores = torch.mul(seq_output, test_item_emb).sum(dim=1)  # [B]
+        return scores
+
+    def forward(self, item_seq, item_seq_len):
+        x_u = self.item_embedding(item_seq).to(self.device)  # [B, N, D]
+
+        # concept activation
+        # sort by inner product
+        x = torch.matmul(x_u, self.w1)
+        x = torch.tanh(x)
+        x = torch.matmul(x, self.w2)
+        a = F.softmax(x, dim=1)
+        z_u = torch.matmul(a.unsqueeze(2).transpose(1, 2), x_u).transpose(1, 2)
+        s_u = torch.matmul(self.C.weight, z_u)
+        s_u = s_u.squeeze(2)
+        idx = s_u.argsort(1)[:, -self.k:]
+        s_u_idx = s_u.sort(1)[0][:, -self.k:]
+        c_u = self.C(idx)
+        sigs = torch.sigmoid(s_u_idx.unsqueeze(2).repeat(1, 1, self.embedding_size))
+        C_u = c_u.mul(sigs)
+
+        # intention assignment
+        # use matrix multiplication instead of cos()
+        w3_x_u_norm = F.normalize(x_u.matmul(self.w3), p=2, dim=2)
+        C_u_norm = self.ln2(C_u)
+        P_k_t = torch.bmm(w3_x_u_norm, C_u_norm.transpose(1, 2))
+        P_k_t_b = F.softmax(P_k_t, dim=2)
+        P_k_t_b_t = P_k_t_b.transpose(1, 2)
+
+        # attention weighting
+        a_k = x_u.unsqueeze(1).repeat(1, self.k, 1, 1).matmul(self.w_k_1)
+        P_t_k = F.softmax(torch.tanh(a_k).matmul(self.w_k_2.reshape(self.k, self.embedding_size, 1)).squeeze(3), dim=2)
+
+        # interest embedding generation
+        mul_p = P_k_t_b_t.mul(P_t_k)
+        x_u_re = x_u.unsqueeze(1).repeat(1, self.k, 1, 1)
+        mul_p_re = mul_p.unsqueeze(3)
+        delta_k = x_u_re.mul(mul_p_re).sum(2)
+        delta_k = F.normalize(delta_k, p=2, dim=2)
+
+        # prototype sequence
+        x_u_bar = P_k_t_b.matmul(C_u)
+        C_apt = F.softmax(torch.tanh(x_u_bar.matmul(self.w3)).matmul(self.w4), dim=1)
+        C_apt = C_apt.reshape(-1, 1, self.max_seq_length).matmul(x_u_bar)
+        C_apt = self.ln4(C_apt)
+
+        # aggregation weight
+        e_k = delta_k.bmm(C_apt.reshape(-1, self.embedding_size, 1)) / self.tau
+        e_k_u = F.softmax(e_k.squeeze(2), dim=1)
+        v_u = e_k_u.unsqueeze(2).mul(delta_k).sum(dim=1)
+
+        return v_u
+
+    def full_sort_predict(self, interaction):
+        item_seq = interaction[self.ITEM_SEQ]
+        item_seq_len = interaction[self.ITEM_SEQ_LEN]
+        seq_output = self.forward(item_seq, item_seq_len)
+        test_items_emb = self.item_embedding.weight
+        scores = torch.matmul(seq_output, test_items_emb.transpose(0, 1))  # [B, n_items]
+        return scores
diff --git a/recbole/properties/model/ADMMSLIM.yaml b/recbole/properties/model/ADMMSLIM.yaml
new file mode 100644
index 000000000..84bec809c
--- /dev/null
+++ b/recbole/properties/model/ADMMSLIM.yaml
@@ -0,0 +1,7 @@
+lambda1: 3
+lambda2: 200
+alpha: 0.50
+rho: 4000
+k: 100
+positive_only: True
+center_columns: False 
\ No newline at end of file
diff --git a/recbole/properties/model/LightSANs.yaml b/recbole/properties/model/LightSANs.yaml
new file mode 100644
index 000000000..71589d80e
--- /dev/null
+++ b/recbole/properties/model/LightSANs.yaml
@@ -0,0 +1,11 @@
+n_layers: 2
+n_heads: 2
+k_interests: 5
+hidden_size: 64
+inner_size: 256
+hidden_dropout_prob: 0.2
+attn_dropout_prob: 0.2
+hidden_act: 'gelu'
+layer_norm_eps: 1e-12
+initializer_range: 0.02
+loss_type: 'CE'
\ No newline at end of file
diff --git a/recbole/properties/model/SGL.yaml b/recbole/properties/model/SGL.yaml
new file mode 100644
index 000000000..1b927702e
--- /dev/null
+++ b/recbole/properties/model/SGL.yaml
@@ -0,0 +1,7 @@
+type: "ED"
+n_layers: 3
+ssl_tau: 0.5
+reg_weight: 1e-5
+ssl_weight: 0.05
+drop_ratio: 0.1
+embedding_size: 64
\ No newline at end of file
diff --git a/recbole/properties/model/SINE.yaml b/recbole/properties/model/SINE.yaml
new file mode 100644
index 000000000..3aa8f688b
--- /dev/null
+++ b/recbole/properties/model/SINE.yaml
@@ -0,0 +1,7 @@
+embedding_size: 64
+layer_norm_eps: 1e-12
+prototype_size: 50
+interest_size: 3
+tau_ratio: 0.1
+reg_loss_ratio: 0.5
+loss_type: 'CE'
diff --git a/recbole/properties/overall.yaml b/recbole/properties/overall.yaml
index 1d8151507..984ce1a4a 100644
--- a/recbole/properties/overall.yaml
+++ b/recbole/properties/overall.yaml
@@ -8,7 +8,9 @@ data_path: 'dataset/'
 checkpoint_dir: 'saved'
 show_progress: True
 save_dataset: False
+dataset_save_path: ~
 save_dataloaders: False
+dataloaders_save_path: ~
 log_wandb: False
 wandb_project: 'recbole'
 
@@ -24,6 +26,7 @@ stopping_step: 10
 clip_grad_norm: ~
 # clip_grad_norm:  {'max_norm': 5, 'norm_type': 2}
 weight_decay: 0.0
+loss_decimal_place: 4
 require_pow: False
 
 # evaluation settings
@@ -38,5 +41,4 @@ topk: [10]
 valid_metric: MRR@10
 valid_metric_bigger: True
 eval_batch_size: 4096
-loss_decimal_place: 4
 metric_decimal_place: 4
\ No newline at end of file
diff --git a/recbole/quick_start/quick_start.py b/recbole/quick_start/quick_start.py
index 05ad69129..e74c69059 100644
--- a/recbole/quick_start/quick_start.py
+++ b/recbole/quick_start/quick_start.py
@@ -39,16 +39,13 @@ def run_recbole(model=None, dataset=None, config_file_list=None, config_dict=Non
 
     # dataset filtering
     dataset = create_dataset(config)
-    if config['save_dataset']:
-        dataset.save()
     logger.info(dataset)
 
     # dataset splitting
     train_data, valid_data, test_data = data_preparation(config, dataset)
-    if config['save_dataloaders']:
-        save_split_dataloaders(config, dataloaders=(train_data, valid_data, test_data))
 
     # model loading and initialization
+    init_seed(config['seed'], config['reproducibility'])
     model = get_model(config['model'])(config, train_data.dataset).to(config['device'])
     logger.info(model)
 
@@ -88,6 +85,7 @@ def objective_function(config_dict=None, config_file_list=None, saved=True):
     logging.basicConfig(level=logging.ERROR)
     dataset = create_dataset(config)
     train_data, valid_data, test_data = data_preparation(config, dataset)
+    init_seed(config['seed'], config['reproducibility'])
     model = get_model(config['model'])(config, train_data.dataset).to(config['device'])
     trainer = get_trainer(config['MODEL_TYPE'], config['model'])(config, model)
     best_valid_score, best_valid_result = trainer.fit(train_data, valid_data, verbose=False, saved=saved)
@@ -101,25 +99,11 @@ def objective_function(config_dict=None, config_file_list=None, saved=True):
     }
 
 
-def load_data_and_model(model_file, dataset_file=None, dataloader_file=None):
+def load_data_and_model(model_file):
     r"""Load filtered dataset, split dataloaders and saved model.
 
     Args:
         model_file (str): The path of saved model file.
-        dataset_file (str, optional): The path of filtered dataset. Defaults to ``None``.
-        dataloader_file (str, optional): The path of split dataloaders. Defaults to ``None``.
-
-    Note:
-        The :attr:`dataset` will be loaded or created according to the following strategy:
-        If :attr:`dataset_file` is not ``None``, the :attr:`dataset` will be loaded from :attr:`dataset_file`.
-        If :attr:`dataset_file` is ``None`` and :attr:`dataloader_file` is ``None``,
-        the :attr:`dataset` will be created according to :attr:`config`.
-        If :attr:`dataset_file` is ``None`` and :attr:`dataloader_file` is not ``None``,
-        the :attr:`dataset` will neither be loaded or created.
-
-        The :attr:`dataloader` will be loaded or created according to the following strategy:
-        If :attr:`dataloader_file` is not ``None``, the :attr:`dataloader` will be loaded from :attr:`dataloader_file`.
-        If :attr:`dataloader_file` is ``None``, the :attr:`dataloader` will be created according to :attr:`config`.
 
     Returns:
         tuple:
@@ -132,20 +116,16 @@ def load_data_and_model(model_file, dataset_file=None, dataloader_file=None):
     """
     checkpoint = torch.load(model_file)
     config = checkpoint['config']
+    init_seed(config['seed'], config['reproducibility'])
     init_logger(config)
+    logger = getLogger()
+    logger.info(config)
 
-    dataset = None
-    if dataset_file:
-        with open(dataset_file, 'rb') as f:
-            dataset = pickle.load(f)
-
-    if dataloader_file:
-        train_data, valid_data, test_data = load_split_dataloaders(dataloader_file)
-    else:
-        if dataset is None:
-            dataset = create_dataset(config)
-        train_data, valid_data, test_data = data_preparation(config, dataset)
+    dataset = create_dataset(config)
+    logger.info(dataset)
+    train_data, valid_data, test_data = data_preparation(config, dataset)
 
+    init_seed(config['seed'], config['reproducibility'])
     model = get_model(config['model'])(config, train_data.dataset).to(config['device'])
     model.load_state_dict(checkpoint['state_dict'])
     model.load_other_parameter(checkpoint.get('other_parameter'))
diff --git a/recbole/trainer/hyper_tuning.py b/recbole/trainer/hyper_tuning.py
index dda0c49e5..ae902a8f7 100644
--- a/recbole/trainer/hyper_tuning.py
+++ b/recbole/trainer/hyper_tuning.py
@@ -95,7 +95,7 @@ def exhaustive_search(new_ids, domain, trials, seed, nbMaxSucessiveFailures=1000
         ) for trial in trials.trials
     ])
 
-    rng = np.random.RandomState(seed)
+    rng = np.random.default_rng(seed)
     rval = []
     for _, new_id in enumerate(new_ids):
         newSample = False
diff --git a/recbole/trainer/trainer.py b/recbole/trainer/trainer.py
index df005ae39..8e66a4eef 100644
--- a/recbole/trainer/trainer.py
+++ b/recbole/trainer/trainer.py
@@ -101,40 +101,52 @@ def __init__(self, config, model):
         self.best_valid_score = -np.inf if self.valid_metric_bigger else np.inf
         self.best_valid_result = None
         self.train_loss_dict = dict()
-        self.optimizer = self._build_optimizer(self.model.parameters())
+        self.optimizer = self._build_optimizer()
         self.eval_type = config['eval_type']
         self.eval_collector = Collector(config)
         self.evaluator = Evaluator(config)
         self.item_tensor = None
         self.tot_item_num = None
 
-    def _build_optimizer(self, params):
+    def _build_optimizer(self, **kwargs):
         r"""Init the Optimizer
 
+        Args:
+            params (torch.nn.Parameter, optional): The parameters to be optimized.
+                Defaults to ``self.model.parameters()``.
+            learner (str, optional): The name of used optimizer. Defaults to ``self.learner``.
+            learning_rate (float, optional): Learning rate. Defaults to ``self.learning_rate``.
+            weight_decay (float, optional): The L2 regularization weight. Defaults to ``self.weight_decay``.
+
         Returns:
             torch.optim: the optimizer
         """
-        if self.config['reg_weight'] and self.weight_decay and self.weight_decay * self.config['reg_weight'] > 0:
+        params = kwargs.pop('params', self.model.parameters())
+        learner = kwargs.pop('learner', self.learner)
+        learning_rate = kwargs.pop('learning_rate', self.learning_rate)
+        weight_decay = kwargs.pop('weight_decay', self.weight_decay)
+
+        if self.config['reg_weight'] and weight_decay and weight_decay * self.config['reg_weight'] > 0:
             self.logger.warning(
                 'The parameters [weight_decay] and [reg_weight] are specified simultaneously, '
                 'which may lead to double regularization.'
             )
 
-        if self.learner.lower() == 'adam':
-            optimizer = optim.Adam(params, lr=self.learning_rate, weight_decay=self.weight_decay)
-        elif self.learner.lower() == 'sgd':
-            optimizer = optim.SGD(params, lr=self.learning_rate, weight_decay=self.weight_decay)
-        elif self.learner.lower() == 'adagrad':
-            optimizer = optim.Adagrad(params, lr=self.learning_rate, weight_decay=self.weight_decay)
-        elif self.learner.lower() == 'rmsprop':
-            optimizer = optim.RMSprop(params, lr=self.learning_rate, weight_decay=self.weight_decay)
-        elif self.learner.lower() == 'sparse_adam':
-            optimizer = optim.SparseAdam(params, lr=self.learning_rate)
-            if self.weight_decay > 0:
+        if learner.lower() == 'adam':
+            optimizer = optim.Adam(params, lr=learning_rate, weight_decay=weight_decay)
+        elif learner.lower() == 'sgd':
+            optimizer = optim.SGD(params, lr=learning_rate, weight_decay=weight_decay)
+        elif learner.lower() == 'adagrad':
+            optimizer = optim.Adagrad(params, lr=learning_rate, weight_decay=weight_decay)
+        elif learner.lower() == 'rmsprop':
+            optimizer = optim.RMSprop(params, lr=learning_rate, weight_decay=weight_decay)
+        elif learner.lower() == 'sparse_adam':
+            optimizer = optim.SparseAdam(params, lr=learning_rate)
+            if weight_decay > 0:
                 self.logger.warning('Sparse Adam cannot argument received argument [{weight_decay}]')
         else:
             self.logger.warning('Received unrecognized optimizer, set default Adam optimizer')
-            optimizer = optim.Adam(params, lr=self.learning_rate)
+            optimizer = optim.Adam(params, lr=learning_rate)
         return optimizer
 
     def _train_epoch(self, train_data, epoch_idx, loss_func=None, show_progress=False):
@@ -198,13 +210,14 @@ def _valid_epoch(self, valid_data, show_progress=False):
         valid_score = calculate_valid_score(valid_result, self.valid_metric)
         return valid_score, valid_result
 
-    def _save_checkpoint(self, epoch):
+    def _save_checkpoint(self, epoch, verbose=True, **kwargs):
         r"""Store the model parameters information and training information.
 
         Args:
             epoch (int): the current epoch id
 
         """
+        saved_model_file = kwargs.pop('saved_model_file', self.saved_model_file)
         state = {
             'config': self.config,
             'epoch': epoch,
@@ -214,7 +227,9 @@ def _save_checkpoint(self, epoch):
             'other_parameter': self.model.other_parameter(),
             'optimizer': self.optimizer.state_dict(),
         }
-        torch.save(state, self.saved_model_file)
+        torch.save(state, saved_model_file)
+        if verbose:
+            self.logger.info(set_color('Saving current', 'blue') + f': {saved_model_file}')
 
     def resume_checkpoint(self, resume_file):
         r"""Load the model parameters information and training information.
@@ -307,9 +322,11 @@ def fit(self, train_data, valid_data=None, verbose=True, saved=True, show_progre
              (float, dict): best valid score and best valid result. If valid_data is None, it returns (-1, None)
         """
         if saved and self.start_epoch >= self.epochs:
-            self._save_checkpoint(-1)
+            self._save_checkpoint(-1, verbose=verbose)
 
         self.eval_collector.data_collect(train_data)
+        if self.config['train_neg_sample_args'].get('dynamic', 'none') != 'none':
+            train_data.get_model(self.model)
         valid_step = 0
 
         for epoch_idx in range(self.start_epoch, self.epochs):
@@ -328,10 +345,7 @@ def fit(self, train_data, valid_data=None, verbose=True, saved=True, show_progre
             # eval
             if self.eval_step <= 0 or not valid_data:
                 if saved:
-                    self._save_checkpoint(epoch_idx)
-                    update_output = set_color('Saving current', 'blue') + ': %s' % self.saved_model_file
-                    if verbose:
-                        self.logger.info(update_output)
+                    self._save_checkpoint(epoch_idx, verbose=verbose)
                 continue
             if (epoch_idx + 1) % self.eval_step == 0:
                 valid_start_time = time()
@@ -356,10 +370,7 @@ def fit(self, train_data, valid_data=None, verbose=True, saved=True, show_progre
 
                 if update_flag:
                     if saved:
-                        self._save_checkpoint(epoch_idx)
-                        update_output = set_color('Saving current best', 'blue') + ': %s' % self.saved_model_file
-                        if verbose:
-                            self.logger.info(update_output)
+                        self._save_checkpoint(epoch_idx, verbose=verbose)
                     self.best_valid_result = valid_result
 
                 if callback_fn:
@@ -428,16 +439,13 @@ def evaluate(self, eval_data, load_best_model=True, model_file=None, show_progre
             show_progress (bool): Show the progress of evaluate epoch. Defaults to ``False``.
 
         Returns:
-            dict: eval result, key is the eval metric and value in the corresponding metric value.
+            collections.OrderedDict: eval result, key is the eval metric and value in the corresponding metric value.
         """
         if not eval_data:
             return
 
         if load_best_model:
-            if model_file:
-                checkpoint_file = model_file
-            else:
-                checkpoint_file = self.saved_model_file
+            checkpoint_file = model_file or self.saved_model_file
             checkpoint = torch.load(checkpoint_file)
             self.model.load_state_dict(checkpoint['state_dict'])
             self.model.load_other_parameter(checkpoint.get('other_parameter'))
@@ -810,9 +818,6 @@ def fit(self, train_data, valid_data=None, verbose=True, saved=True, show_progre
                     if saved:
                         self.model.save_model(self.temp_best_file)
                         self._save_checkpoint(epoch_idx)
-                        update_output = set_color('Saving current best', 'blue') + ': %s' % self.saved_model_file
-                        if verbose:
-                            self.logger.info(update_output)
                     self.best_valid_result = valid_result
 
                 if stop_flag:
@@ -1021,8 +1026,8 @@ def __init__(self, config, model):
         self.n_enc_epochs = config['n_enc_epochs']
         self.n_dec_epochs = config['n_dec_epochs']
 
-        self.optimizer_encoder = self._build_optimizer(self.model.encoder.parameters())
-        self.optimizer_decoder = self._build_optimizer(self.model.decoder.parameters())
+        self.optimizer_encoder = self._build_optimizer(params=self.model.encoder.parameters())
+        self.optimizer_decoder = self._build_optimizer(params=self.model.decoder.parameters())
 
     def _train_epoch(self, train_data, epoch_idx, loss_func=None, show_progress=False):
         self.optimizer = self.optimizer_encoder
diff --git a/recbole/utils/argument_list.py b/recbole/utils/argument_list.py
index e49ed2509..655f51b08 100644
--- a/recbole/utils/argument_list.py
+++ b/recbole/utils/argument_list.py
@@ -10,31 +10,31 @@
     'reproducibility',
     'state',
     'data_path',
-    'benchmark_filename',
+    'checkpoint_dir',
     'show_progress',
     'config_file',
     'save_dataset',
+    'dataset_save_path',
     'save_dataloaders',
+    'dataloaders_save_path',
     'log_wandb',
 ]
 
 training_arguments = [
     'epochs', 'train_batch_size',
     'learner', 'learning_rate',
-    'training_neg_sample_num',
-    'training_neg_sample_distribution',
+    'neg_sampling',
     'eval_step', 'stopping_step',
-    'checkpoint_dir',
     'clip_grad_norm',
+    'weight_decay',
     'loss_decimal_place',
-    'weight_decay'
 ]
 
 evaluation_arguments = [
-    'eval_args',
+    'eval_args', 'repeatable',
     'metrics', 'topk', 'valid_metric', 'valid_metric_bigger',
     'eval_batch_size',
-    'metric_decimal_place'
+    'metric_decimal_place',
 ]
 
 dataset_arguments = [
@@ -46,9 +46,9 @@
     'ITEM_LIST_LENGTH_FIELD', 'LIST_SUFFIX', 'MAX_ITEM_LIST_LENGTH', 'POSITION_FIELD',
     'HEAD_ENTITY_ID_FIELD', 'TAIL_ENTITY_ID_FIELD', 'RELATION_ID_FIELD', 'ENTITY_ID_FIELD',
     'load_col', 'unload_col', 'unused_col', 'additional_feat_suffix',
-    'filter_inter_by_user_or_item', 'rm_dup_inter',
-    'val_interval', 'user_inter_num_interval', 'item_inter_num_interval',
+    'rm_dup_inter', 'val_interval', 'filter_inter_by_user_or_item',
+    'user_inter_num_interval', 'item_inter_num_interval',
     'alias_of_user_id', 'alias_of_item_id', 'alias_of_entity_id', 'alias_of_relation_id',
-    'preload_weight',
-    'normalize_field', 'normalize_all'
+    'preload_weight', 'normalize_field', 'normalize_all',
+    'benchmark_filename',
 ]
diff --git a/recbole/utils/case_study.py b/recbole/utils/case_study.py
index 35bbf4a5d..17af387d8 100644
--- a/recbole/utils/case_study.py
+++ b/recbole/utils/case_study.py
@@ -57,7 +57,7 @@ def full_sort_scores(uid_series, model, test_data, device=None):
     try:
         scores = model.full_sort_predict(input_interaction)
     except NotImplementedError:
-        input_interaction = input_interaction.repeat(dataset.item_num)
+        input_interaction = input_interaction.repeat_interleave(dataset.item_num)
         input_interaction.update(test_data.dataset.get_item_feature().to(device).repeat(len(uid_series)))
         scores = model.predict(input_interaction)
 
diff --git a/recbole/utils/utils.py b/recbole/utils/utils.py
index 02c4714df..212005363 100644
--- a/recbole/utils/utils.py
+++ b/recbole/utils/utils.py
@@ -121,7 +121,7 @@ def early_stopping(value, best, cur_step, max_step, bigger=True):
     stop_flag = False
     update_flag = False
     if bigger:
-        if value > best:
+        if value >= best:
             cur_step = 0
             best = value
             update_flag = True
@@ -130,7 +130,7 @@ def early_stopping(value, best, cur_step, max_step, bigger=True):
             if cur_step > max_step:
                 stop_flag = True
     else:
-        if value < best:
+        if value <= best:
             cur_step = 0
             best = value
             update_flag = True
@@ -167,10 +167,7 @@ def dict2str(result_dict):
         str: result str
     """
 
-    result_str = ''
-    for metric, value in result_dict.items():
-        result_str += str(metric) + ' : ' + str(value) + '    '
-    return result_str
+    return '    '.join([str(metric) + ' : ' + str(value) for metric, value in result_dict.items()])
 
 
 def init_seed(seed, reproducibility):
diff --git a/run_hyper.py b/run_hyper.py
index 8849097be..c0e422aae 100644
--- a/run_hyper.py
+++ b/run_hyper.py
@@ -18,6 +18,7 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('--config_files', type=str, default=None, help='fixed config files')
     parser.add_argument('--params_file', type=str, default=None, help='parameters file')
+    parser.add_argument('--output_file', type=str, default='hyper_example.result', help='output file')
     args, _ = parser.parse_known_args()
 
     # plz set algo='exhaustive' to use exhaustive search, in this case, max_evals is auto set
@@ -25,7 +26,7 @@ def main():
     hp = HyperTuning(objective_function, algo='exhaustive',
                      params_file=args.params_file, fixed_config_file_list=config_file_list)
     hp.run()
-    hp.export_result(output_file='hyper_example.result')
+    hp.export_result(output_file=args.output_file)
     print('best params: ', hp.best_params)
     print('best result: ')
     print(hp.params2result[hp.params2str(hp.best_params)])


Test patch:
diff --git a/asset/time_test_result/Sequential_recommendation.md b/asset/time_test_result/Sequential_recommendation.md
index 289a5e13c..3018a70f2 100644
--- a/asset/time_test_result/Sequential_recommendation.md
+++ b/asset/time_test_result/Sequential_recommendation.md
@@ -42,6 +42,7 @@ Machine Specs:        14 CPU machine, 256GB RAM
 | GCSAN            |                   671.66  |                       4.99  |                    2.96  |
 | KSR              |                    61.18  |                       0.21  |                    6.94  |
 | GRU4RecKG        |                    11.36  |                       0.07  |                    2.22  |
+| LightSANs        |                    28.21  |                       0.12  |                    3.85  |
 
 #### Config file of ml-1m dataset:
 
@@ -117,6 +118,7 @@ load_col:
 | GCSAN            |                    69.03  |                      14.29  |                    3.13  |
 | KSR              |                         - |                           - |                        - |
 | GRU4RecKG        |                         - |                           - |                        - |
+| LightSANs        |                     7.12  |                       0.88  |                    3.48  |
 
 #### Config file of DIGINETICA dataset:
 
@@ -183,14 +185,15 @@ load_col:
 | NextItNet        |                   413.43  |                       5.57  |                    5.73  |
 | TransRec         |                         - |                           - |       CUDA out of memory |
 | S3Rec            |                         - |                           - |       CUDA out of memory |
-| GRU4RecF         |                     62,41 |                        1.32 |                     7.36 |
-| SASRecF          |                    86.20  |                        1.42 |                     6.12 |
+| GRU4RecF         |                    62,41  |                       1.32  |                    7.36  |
+| SASRecF          |                    86.20  |                       1.42  |                    6.12  |
 | BERT4Rec         |                         - |                           - |       CUDA out of memory |
 | FDSA             |                   133.52  |                       1.87  |                    9.19  |
 | SRGNN            |                  1165.06  |                      30.35  |                    3.41  |
 | GCSAN            |                  1112.46  |                      29.84  |                    4.67  |
 | KSR              |                         - |                           - |                        - |
 | GRU4RecKG        |                         - |                           - |                        - |
+| LightSANs        |                    58.46  |                       0.83  |                    5.08  |
 
 #### Config file of Yelp dataset:
 
diff --git a/run_test.sh b/run_test.sh
index e035d878e..03c6406ba 100644
--- a/run_test.sh
+++ b/run_test.sh
@@ -7,7 +7,11 @@ echo "metrics tests finished"
 python -m pytest -v tests/config/test_config.py
 python -m pytest -v tests/config/test_overall.py
 export PYTHONPATH=.
-python tests/config/test_command_line.py --use_gpu=False --valid_metric=Recall@10  --metrics=['Recall@10'] --epochs=200  --learning_rate=0.3
+python tests/config/test_command_line.py --use_gpu=False --valid_metric=Recall@10  --metrics="['Recall']" --epochs=200 --learning_rate=0.3
+if [ $? -ne 0 ]
+then
+  echo -e "\033[31mFailed in test_command_line.py\033[0m"
+fi
 echo "config tests finished"
 
 python -m pytest -v tests/evaluation_setting
diff --git a/tests/config/test_command_line.py b/tests/config/test_command_line.py
index c4ea6ac01..d8916f1ee 100644
--- a/tests/config/test_command_line.py
+++ b/tests/config/test_command_line.py
@@ -13,21 +13,12 @@
 
 if __name__ == '__main__':
 
-    parameters_dict = {
-        'model': 'SASRec',
-        'learning_rate': 0.2,
-        'topk': [50, 100],
-        'epochs': 100,
-    }
-
-    config_file_list = ['test_config_example.yaml']
-
     config = Config(model='BPR', dataset='ml-100k')
 
     # command line
-    assert config['use_gpu'] == False
+    assert config['use_gpu'] is False
     assert config['valid_metric'] == 'Recall@10'
-    # assert config['metrics'] == ['Recall@10']     # bug
+    assert config['metrics'] == ['Recall']     # bug
 
     # priority
     assert config['epochs'] == 200
diff --git a/tests/model/test_model_auto.py b/tests/model/test_model_auto.py
index 17360cd6f..532d9da78 100644
--- a/tests/model/test_model_auto.py
+++ b/tests/model/test_model_auto.py
@@ -41,6 +41,16 @@ def test_bpr(self):
         }
         quick_test(config_dict)
 
+    def test_bpr_with_dns(self):
+        config_dict = {
+            'model': 'BPR',
+            'neg_sampling': {
+                'uniform': 1,
+                'dynamic': 2
+            }
+        }
+        quick_test(config_dict)
+
     def test_neumf(self):
         config_dict = {
             'model': 'NeuMF',
@@ -199,6 +209,18 @@ def test_slimelastic(self):
             'model': 'SLIMElastic',
         }
         quick_test(config_dict)
+    
+    def test_SGL(self):
+        config_dict = {
+            'model': 'SGL',
+        }
+        quick_test(config_dict)
+    
+    def test_ADMMSLIM(self):
+        config_dict = {
+            'model': 'ADMMSLIM',
+        }
+        quick_test(config_dict)
 
 
 class TestContextRecommender(unittest.TestCase):
@@ -656,8 +678,7 @@ def test_bert4rec(self):
             'model': 'BERT4Rec',
             'neg_sampling': None
         }
-        objective_function(config_dict=config_dict,
-                           config_file_list=config_file_list, saved=False)
+        quick_test(config_dict)
 
     def test_bert4rec_with_BPR_loss_and_swish(self):
         config_dict = {
@@ -665,8 +686,43 @@ def test_bert4rec_with_BPR_loss_and_swish(self):
             'loss_type': 'BPR',
             'hidden_act': 'swish'
         }
-        objective_function(config_dict=config_dict,
-                           config_file_list=config_file_list, saved=False)
+        quick_test(config_dict)
+
+    def test_lightsans(self):
+        config_dict = {
+            'model': 'LightSANs',
+            'neg_sampling': None
+        }
+        quick_test(config_dict)
+
+    def test_lightsans_with_BPR_loss(self):
+        config_dict = {
+            'model': 'LightSANs',
+            'loss_type': 'BPR',
+        }
+        quick_test(config_dict)
+
+    def test_sine(self):
+        config_dict = {
+            'model': 'SINE',
+            'neg_sampling': None
+        }
+        quick_test(config_dict)
+
+    def test_sine_with_BPR_loss(self):
+        config_dict = {
+            'model': 'SINE',
+            'loss_type': 'BPR',
+        }
+        quick_test(config_dict)
+
+    def test_sine_with_NLL_loss(self):
+        config_dict = {
+            'model': 'SINE',
+            'neg_sampling': None,
+            'loss_type': 'NLL',
+        }
+        quick_test(config_dict)
 
     # def test_gru4reckg(self):
     #     config_dict = {