init

.gitattributes

@@ -29,3 +29,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zstandard filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Ali Pourramezan Fard
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,77 @@
----
-license: mit
----
+# [ACR-Loss](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=96lS6HIAAAAJ&citation_for_view=96lS6HIAAAAJ:eQOLeE2rZwMC)
+
+### Accepted in ICPR 2022
+ACR Loss: Adaptive Coordinate-based Regression Loss for Face Alignment
+
+#### Link to the paper:
+https://arxiv.org/pdf/2203.15835.pdf
+
+
+```diff
+@@plaese STAR the repo if you like it.@@
+```
+
+```
+Please cite this work as:
+
+@article{fard2022acr,
+  title={ACR Loss: Adaptive Coordinate-based Regression Loss for Face Alignment},
+  author={Fard, Ali Pourramezan and Mahoor, Mohammah H},
+  journal={arXiv preprint arXiv:2203.15835},
+  year={2022}
+}
+```
+
+![Samples](https://github.com/aliprf/ACR-Loss/blob/main/img/ACR_300w_samples.png?raw=true)
+
+## Introduction
+
+Although deep neural networks have achieved reasonable accuracy in solving face alignment, it is still a challenging task, specifically when we deal with facial images, under occlusion, or extreme head poses. Heatmap-based Regression (HBR) and Coordinate-based Regression (CBR) are among the two mainly used methods for face alignment. CBR methods require less computer memory, though their performance is less than HBR methods. In this paper, we propose an Adaptive Coordinatebased Regression (ACR) loss to improve the accuracy of CBR for face alignment. Inspired by the Active Shape Model (ASM), we generate Smooth-Face objects, a set of facial landmark points with less variations compared to the ground truth landmark points. We then introduce a method to estimate the level of difficulty in predicting each landmark point for the network by comparing the distribution of the ground truth landmark points
+and the corresponding Smooth-Face objects. Our proposed ACR Loss can adaptively modify its curvature and the influence of the loss based on the difficulty level of predicting each landmark point in a face. Accordingly, the ACR Loss guides the network toward challenging points than easier points, which improves the accuracy of the face alignment task. Our extensive evaluation shows the capabilities of the proposed ACR Loss in predicting facial landmark points in various facial images.
+
+We evaluated our ACR Loss using MobileNetV2, EfficientNetB0, and EfficientNet-B3 on widely used 300W, and COFW datasets and showed that the performance of face alignment using the ACR Loss is much better than the widely-used L2 loss. Moreover, on the COFW dataset, we achieved state-of-theart accuracy. In addition, on 300W the ACR Loss performance is comparable to the state-of-the-art methods. We also compared the performance of MobileNetV2 trained using the ACR Loss with the lightweight state-of-the-art methods, and we achieved the best accuracy, highlighting the effectiveness of our ACR Loss for face alignment specifically for the lightweight models.
+
+
+----------------------------------------------------------------------------------------------------------------------------------
+## Installing the requirements
+In order to run the code you need to install python >= 3.5. 
+The requirements and the libraries needed to run the code can be installed using the following command:
+
+```
+  pip install -r requirements.txt
+```
+
+
+## Using the pre-trained models
+You can test and use the preetrained models using the following codes:  
+```
+ tester = Test()
+    tester.test_model(ds_name=DatasetName.w300,
+                      pretrained_model_path='./pre_trained_models/ACRLoss/300w/EF_3/300w_EF3_ACRLoss.h5')
+
+```
+
+
+## Training Network from scratch
+
+
+### Preparing Data
+Data needs to be normalized and saved in npy format. 
+
+### PCA creation
+you can you the pca_utility.py class to create the eigenvalues, eigenvectors, and the meanvector:
+```
+pca_calc = PCAUtility()
+    pca_calc.create_pca_from_npy(dataset_name=DatasetName.w300,
+                                 labels_npy_path='./data/w300/normalized_labels/',
+                                 pca_percentages=90)
+
+```
+### Training 
+The training implementation is located in train.py class. You can use the following code to start the training:
+
+```
+    trainer = Train(arch=ModelArch.MNV2,
+                    dataset_name=DatasetName.w300,
+                    save_path='./')
+```

acr_loss.py

@@ -0,0 +1,22 @@
+from config import DatasetName
+import tensorflow as tf
+
+
+class ACRLoss:
+    def acr_loss(self, x_pr, x_gt, phi, lambda_weight, ds_name):
+        low_map = tf.cast(tf.abs(x_pr - x_gt) <= 1.0, dtype=tf.float32)
+        high_map = tf.cast(tf.abs(x_pr - x_gt) > 1.0, dtype=tf.float32)
+
+        '''Big errors'''
+        ln_2 = tf.ones_like(x_pr, dtype=tf.float32) * tf.math.log(2.0)
+        C = tf.cast(tf.cast(phi, dtype=tf.double) * tf.cast(ln_2, dtype=tf.double) - 1.0, dtype=tf.float32)
+        loss_high = 100 * tf.reduce_mean(tf.math.multiply(high_map, (tf.square(x_pr - x_gt) + C)))
+
+        '''Small errors'''
+        power = tf.cast(2.0 - phi, tf.dtypes.float32)
+        ll = tf.pow(tf.abs(x_pr - x_gt), power)
+        loss_low = 100 * tf.reduce_mean(tf.math.multiply(low_map, (lambda_weight * tf.math.log(1.0 + ll))))
+
+        loss_total = loss_low + loss_high
+
+        return loss_total, loss_low, loss_high

cnn.py

@@ -0,0 +1,104 @@
+from config import DatasetName, DatasetType, W300Conf, InputDataSize, LearningConfig
+import tensorflow as tf
+from tensorflow import keras
+from skimage.transform import resize
+from keras.regularizers import l2, l1
+from keras.models import Model
+from keras.applications import mobilenet_v2, mobilenet, densenet
+from keras.layers import Dense, MaxPooling2D, Conv2D, Flatten, \
+    BatchNormalization, Activation, GlobalAveragePooling2D, DepthwiseConv2D, Dropout, ReLU, Concatenate, Input, \
+    GlobalMaxPool2D
+import efficientnet.tfkeras as efn
+
+
+class CNNModel:
+    def get_model(self, arch, output_len):
+
+        if arch == 'mobileNetV2':
+            model = self.create_MobileNet(inp_shape=[224, 224, 3], output_len=output_len)
+
+        elif arch == 'efNb0':
+            model = self.create_efficientNet_b0(inp_shape=[224, 224, 3], input_tensor=None, output_len=output_len)
+
+        elif arch == 'efNb3':
+            model = self.create_efficientNet_b3(inp_shape=[224, 224, 3], input_tensor=None, output_len=output_len)
+
+        return
+
+    def create_MobileNet(self, inp_shape, output_len):
+        mobilenet_model = mobilenet_v2.MobileNetV2(input_shape=inp_shape,
+                                                   alpha=1.0,
+                                                   include_top=True,
+                                                   weights=None,
+                                                   input_tensor=None,
+                                                   pooling=None)
+        mobilenet_model.layers.pop()
+
+        x = mobilenet_model.get_layer('global_average_pooling2d').output  # 1280
+        x = Dropout(0.1)(x)
+        out_landmarks = Dense(output_len, activation=keras.activations.linear, kernel_initializer=initializer,
+                              use_bias=True, name='O_L')(x)
+        inp = mobilenet_model.input
+
+        revised_model = Model(inp, [out_landmarks])
+
+        revised_model.summary()
+        model_json = revised_model.to_json()
+
+        with open("mobileNet_v2.json", "w") as json_file:
+            json_file.write(model_json)
+
+        return revised_model
+
+    def create_efficientNet_b0(self, inp_shape, input_tensor, output_len):
+        initializer = tf.keras.initializers.he_uniform()
+
+        eff_net = efn.EfficientNetB0(include_top=True,
+                                     weights=None,
+                                     input_tensor=input_tensor,
+                                     input_shape=inp_shape,
+                                     pooling=None)
+        eff_net.layers.pop()
+        inp = eff_net.input
+
+        x = eff_net.get_layer('top_activation').output
+        x = GlobalAveragePooling2D()(x)
+        x = keras.layers.Dropout(rate=0.5)(x)
+        output = Dense(output_len, activation='linear', use_bias=True, name='out',
+                       kernel_initializer=initializer)(x)
+
+        eff_net = Model(inp, output)
+        eff_net.summary()
+
+        model_json = eff_net.to_json()
+        with open("eff_net_b0.json", "w") as json_file:
+            json_file.write(model_json)
+
+        return eff_net
+
+    def create_efficientNet_b3(self, inp_shape, input_tensor, output_len):
+        initializer = tf.keras.initializers.he_uniform()
+
+        eff_net = efn.EfficientNetB3(include_top=True,
+                                     weights=None,
+                                     input_tensor=input_tensor,
+                                     input_shape=inp_shape,
+                                     pooling=None)
+        eff_net.layers.pop()
+        inp = eff_net.input
+
+        x = eff_net.get_layer('top_activation').output
+        x = GlobalAveragePooling2D()(x)
+        x = keras.layers.Dropout(rate=0.5)(x)
+
+        output = Dense(output_len, activation='linear', use_bias=True, name='out',
+                       kernel_initializer=initializer)(x)
+
+        eff_net = Model(inp, output)
+        eff_net.summary()
+
+        model_json = eff_net.to_json()
+        with open("eff_net_b3.json", "w") as json_file:
+            json_file.write(model_json)
+
+        return eff_net

config.py

@@ -0,0 +1,44 @@
+class DatasetName:
+    w300 = '300W'
+    cofw = 'COFW'
+
+
+class ModelArch:
+    MNV2 = 'mobileNetV2'
+    EFNB0 = 'EfficientNet-B0'
+    EFNB3 = 'EfficientNet-B3'
+
+
+class DatasetType:
+    data_type_train = 0
+    data_type_test = 1
+
+
+class LearningConfig:
+    batch_size = 3
+    epochs = 150
+
+
+class InputDataSize:
+    image_input_size = 224
+
+
+class W300Conf:
+    W300W_prefix_path = './300W/'
+
+    train_annotation = W300W_prefix_path + 'train_set/annotations/'
+    train_image = W300W_prefix_path + 'train_set/images/'
+
+    test_annotation_path = W300W_prefix_path + 'test_set/annotations/'
+    test_image_path = W300W_prefix_path + 'test_set/images/'
+    num_of_landmarks = 68
+
+class CofwConf:
+    cofw_prefix_path = './cofw/'
+
+    train_annotation = cofw_prefix_path + 'train_set/annotations/'
+    train_image = cofw_prefix_path + 'train_set/images/'
+
+    test_annotation_path = cofw_prefix_path + 'test_set/annotations/'
+    test_image_path = cofw_prefix_path + 'test_set/images/'
+    num_of_landmarks = 29

data_util.py

@@ -0,0 +1,52 @@
+from pca_utilities import PCAUtility
+from config import DatasetName, DatasetType, W300Conf, InputDataSize, LearningConfig, CofwConf
+import tensorflow as tf
+import numpy as np
+import os
+from skimage.transform import resize
+import csv
+import sys
+from PIL import Image
+from pathlib import Path
+import sqlite3
+import cv2
+import os.path
+from keras import backend as K
+
+from scipy import misc
+from scipy.ndimage import gaussian_filter, maximum_filter
+from numpy import save, load, asarray
+from tqdm import tqdm
+import pickle
+import PIL.ImageDraw as ImageDraw
+import math
+
+
+class DataUtil:
+
+    def __init__(self, number_of_landmark):
+        self.number_of_landmark = number_of_landmark
+
+    def get_asm(self, input, dataset_name, accuracy, alpha=1.0):
+        pca_utils = PCAUtility()
+
+        eigenvalues = load('pca_obj/' + dataset_name + pca_utils.eigenvalues_prefix + str(accuracy) + ".npy")
+        eigenvectors = load('pca_obj/' + dataset_name + pca_utils.eigenvectors_prefix + str(accuracy) + ".npy")
+        meanvector = load('pca_obj/' + dataset_name + pca_utils.meanvector_prefix + str(accuracy) + ".npy")
+
+        b_vector_p = pca_utils.calculate_b_vector(input, True, eigenvalues, eigenvectors, meanvector)
+        out = alpha * meanvector + np.dot(eigenvectors, b_vector_p)
+        return out
+
+    def create_image_and_labels_name(self, img_path, annotation_path):
+        img_filenames = []
+        lbls_filenames = []
+
+        for file in os.listdir(img_path):
+            if file.endswith(".jpg") or file.endswith(".png"):
+                lbl_file = str(file)[:-3] + "npy"  # just name
+                if os.path.exists(annotation_path + lbl_file):
+                    img_filenames.append(str(file))
+                    lbls_filenames.append(lbl_file)
+
+        return np.array(img_filenames), np.array(lbls_filenames)

main.py

@@ -0,0 +1,24 @@
+from train import Train
+from test import Test
+from config import DatasetName, ModelArch
+from pca_utilities import PCAUtility
+
+if __name__ == '__main__':
+    '''use the pretrained model'''
+    tester = Test()
+    tester.test_model(ds_name=DatasetName.w300,
+                      pretrained_model_path='./pre_trained_models/ACRLoss/mnv2.h5')
+
+    '''training model from scratch'''
+
+    #   pretrain prerequisites
+    #       1- PCA calculation:
+    pca_calc = PCAUtility()
+    pca_calc.create_pca_from_npy(dataset_name=DatasetName.w300,
+                                 labels_npy_path='./data/w300/normalized_labels/',
+                                 pca_percentages=90)
+
+    #  Train:
+    trainer = Train(arch=ModelArch.MNV2,
+                    dataset_name=DatasetName.w300,
+                    save_path='./')

pca_utilities.py

@@ -0,0 +1,76 @@
+from sklearn.decomposition import PCA, IncrementalPCA
+from sklearn.decomposition import TruncatedSVD
+import numpy as np
+import pickle
+import os
+from tqdm import tqdm
+from numpy import save, load
+import math
+from PIL import Image
+from numpy import save, load
+
+
+class PCAUtility:
+    eigenvalues_prefix = "_eigenvalues_"
+    eigenvectors_prefix = "_eigenvectors_"
+    meanvector_prefix = "_meanvector_"
+
+    def create_pca_from_npy(self, dataset_name, labels_npy_path, pca_percentages):
+        """
+        generate and save eigenvalues, eigenvectors, meanvector
+        :param labels_npy_path: the path to the normalized labels that are save in npy format.
+        :param pca_percentages: % of eigenvalues that will be used
+        :return: generate
+        """
+        path = labels_npy_path
+        print('PCA calculation started: loading labels')
+
+        lbl_arr = []
+        for file in tqdm(os.listdir(path)):
+            if file.endswith(".npy"):
+                npy_file = os.path.join(path, file)
+                lbl_arr.append(load(npy_file))
+
+        lbl_arr = np.array(lbl_arr)
+
+        reduced_lbl_arr, eigenvalues, eigenvectors = self._func_PCA(lbl_arr, pca_percentages)
+        mean_lbl_arr = np.mean(lbl_arr, axis=0)
+        eigenvectors = eigenvectors.T
+
+        save('./pca_obj/' + dataset_name + self.eigenvalues_prefix + str(pca_percentages), eigenvalues)
+        save('./pca_obj/' + dataset_name + self.eigenvectors_prefix + str(pca_percentages), eigenvectors)
+        save('./pca_obj/' + dataset_name + self.meanvector_prefix + str(pca_percentages), mean_lbl_arr)
+
+    def load_pca_obj(self, dataset_name, pca_percentages):
+        eigenvalues = np.load('./pca_obj/' + dataset_name + self.eigenvalues_prefix + str(pca_percentages))
+        eigenvectors = np.load('./pca_obj/' + dataset_name + self.eigenvectors_prefix + str(pca_percentages))
+        meanvector = np.load('./pca_obj/' + dataset_name + self.meanvector_prefix + str(pca_percentages))
+        return eigenvalues, eigenvectors, meanvector
+
+    def calculate_b_vector(self, predicted_vector, correction, eigenvalues, eigenvectors, meanvector):
+        tmp1 = predicted_vector - meanvector
+        b_vector = np.dot(eigenvectors.T, tmp1)
+
+        # put b in -3lambda =>
+        if correction:
+            i = 0
+            for b_item in b_vector:
+                lambda_i_sqr = 3 * math.sqrt(eigenvalues[i])
+
+                if b_item > 0:
+                    b_item = min(b_item, lambda_i_sqr)
+                else:
+                    b_item = max(b_item, -1 * lambda_i_sqr)
+                b_vector[i] = b_item
+                i += 1
+
+        return b_vector
+
+    def _func_PCA(self, input_data, pca_postfix):
+        input_data = np.array(input_data)
+        pca = PCA(n_components=pca_postfix / 100)
+        pca.fit(input_data)
+        pca_input_data = pca.transform(input_data)
+        eigenvalues = pca.explained_variance_
+        eigenvectors = pca.components_
+        return pca_input_data, eigenvalues, eigenvectors

pre_trained_models/300w/EF_0/300w_EF0_ACRLoss.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:937fc62cb7517a427905dc5e0c39711615269f481b080eb04b6bc30e2c22d7dd
+size 17518832

pre_trained_models/300w/EF_0/300w_EF0_base.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:55a7836679d92933220c96e0b0563466aa4d24a4866b6ec3e0b5f44714a85e9a
+size 17519192

pre_trained_models/300w/EF_3/300w_EF3_ACRLoss.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abd7b1bc7e43cd4489fbde4f83a4dfb3722596f0dcd3e909256d4720af2b1920
+size 44972528

pre_trained_models/300w/EF_3/300w_EF3_base.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f465c57e83bf4a8ec901baa0995b3d29a83fdf49af13432cdb7ccd99dccc2a53
+size 44972528

pre_trained_models/300w/mnv2/300w_mnv2_ACRLoss.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97eabc914cf24ffe6fd32442748c7c35d12297950feab8c20204b0a2b872d304
+size 10197672

pre_trained_models/300w/mnv2/300w_mnv2_base.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1ac8c5431a516e1d5218e7da3a86ebde77b71c6054d2654d76445d2e5f44ac0
+size 10195168

pre_trained_models/cofw/EF_0/cofw_EF0_base.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90283e68ac497f2baaec7c02c605be3bb8b9eb283fcd452e0c93c3c174c7a476
+size 17119832

pre_trained_models/cofw/EF_0/cofw_EF0_xxxLoss.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c1ff7991fa3b0434e3b75ea32127d77a3bd4c0e3ba241aa54276eb3dff67460
+size 17119832

pre_trained_models/cofw/EF_3/cofw_EF3_base.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36f7006d1215031b737a8072b4c67685308203eea59d431f5073d7687903ea4f
+size 44491224

pre_trained_models/cofw/EF_3/cofw_EF3_xxxLoss.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e495279c4c60de5ad07e1aa29db1a361c51029f260f0cf0ea0a0b73fb260c59e
+size 44491224

pre_trained_models/cofw/mnv2/cofw_mnv2_base.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5002926568613fe55a3e005987e4f9e41ef9859f9dfae9c2a0e2b19bdbc300f6
+size 9799400

pre_trained_models/cofw/mnv2/cofw_mnv2_xxxLoss.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4a2623fc19644604e401fc22c2104f1293645672464927fee72e37ad8609724
+size 9800464

requirements.txt

@@ -0,0 +1,18 @@
+numpy
+
+# Mac M1:
+tensorflow-macos
+#linux
+# tensorflow
+
+keras
+matplotlib
+opencv-python
+opencv-contrib-python
+scipy
+scikit-learn
+scikit-image
+Pillow
+tqdm
+efficientnet
+tensorboard

test.py

@@ -0,0 +1,37 @@
+from config import DatasetName, W300Conf, DatasetType, LearningConfig, InputDataSize, CofwConf
+import tensorflow as tf
+
+import cv2
+import os.path
+import scipy.io as sio
+from cnn import CNNModel
+from tqdm import tqdm
+import numpy as np
+from os import listdir
+from os.path import isfile, join
+from scipy.integrate import simps
+from scipy.integrate import trapz
+import matplotlib.pyplot as plt
+from skimage.io import imread
+
+
+class Test:
+    def test_model(self, pretrained_model_path, ds_name):
+        if ds_name == DatasetName.w300:
+            test_annotation_path = W300Conf.test_annotation_path
+            test_image_path = W300Conf.test_image_path
+        elif ds_name == DatasetName.cofw:
+            test_annotation_path = CofwConf.test_annotation_path
+            test_image_path = CofwConf.test_image_path
+
+        model = tf.keras.models.load_model(pretrained_model_path)
+
+        for i, file in tqdm(enumerate(os.listdir(test_image_path))):
+            # load image and then normalize it
+            img = imread(test_image_path + file) / 255.0
+
+            # prediction
+            prediction = model.predict(np.expand_dims(img, axis=0))
+
+            # the first dimension is landmark point
+            landmark_predicted = prediction

train.py

@@ -0,0 +1,218 @@
+from config import DatasetName, W300Conf, DatasetType, LearningConfig, InputDataSize, CofwConf
+from cnn import CNNModel
+import tensorflow as tf
+from tensorflow import keras
+import numpy as np
+import matplotlib.pyplot as plt
+import math
+from datetime import datetime
+from sklearn.utils import shuffle
+from sklearn.model_selection import train_test_split
+from numpy import save, load, asarray
+import csv
+from skimage.io import imread
+import pickle
+from tqdm import tqdm
+import os
+from data_util import DataUtil
+from acr_loss import ACRLoss
+
+
+class Train:
+    def __init__(self, arch, dataset_name, save_path, lambda_weight):
+
+        self.lambda_weight = lambda_weight
+        self.dataset_name = dataset_name
+        self.save_path = save_path
+        self.arch = arch
+        self.base_lr = 1e-3
+        self.max_lr = 5e-3
+        if dataset_name == DatasetName.w300:
+            self.num_landmark = W300Conf.num_of_landmarks * 2
+            self.img_path = W300Conf.train_image
+            self.annotation_path = W300Conf.train_annotation
+            '''evaluation path:'''
+            self.eval_img_path = W300Conf.test_image_path + 'challenging/'
+            self.eval_annotation_path = W300Conf.test_annotation_path + 'challenging/'
+
+        if dataset_name == DatasetName.cofw:
+            self.num_landmark = CofwConf.num_of_landmarks * 2
+            self.img_path = CofwConf.train_image
+            self.annotation_path = CofwConf.train_annotation
+            '''evaluation path:'''
+            self.eval_img_path = CofwConf.test_image_path
+            self.eval_annotation_path = CofwConf.test_annotation_path
+
+    def train(self, weight_path):
+        """
+        :param weight_path:
+        :return:
+        """
+        '''create loss'''
+        c_loss = ACRLoss()
+
+        '''create summary writer'''
+        summary_writer = tf.summary.create_file_writer(
+            "./train_logs/fit/" + datetime.now().strftime("%Y%m%d-%H%M%S"))
+
+        '''create sample generator'''
+        x_train_filenames, y_train_filenames = self._create_generators()
+
+        '''making models'''
+        model = self.make_model(arch=self.arch, w_path=weight_path)
+
+        '''create train configuration'''
+        step_per_epoch = len(x_train_filenames) // LearningConfig.batch_size
+
+        lr = 1e-3
+        for epoch in range(LearningConfig.epochs):
+            '''calculate Learning rate'''
+            optimizer = self._get_optimizer(lr=lr)
+            ''''''
+            x_train_filenames, y_train_filenames = self._shuffle_data(x_train_filenames, y_train_filenames)
+            for batch_index in range(step_per_epoch):
+                '''load annotation and images'''
+                images, annotation_gr = self._get_batch_sample(
+                    batch_index=batch_index, x_train_filenames=x_train_filenames,
+                    y_train_filenames=y_train_filenames)
+
+                phi = self.calculate_adoptive_weight(epoch=epoch, batch_index=batch_index,
+                                                     y_train_filenames=y_train_filenames,
+                                                     weight_path=weight_path)
+
+                '''convert to tensor'''
+                images = tf.cast(images, tf.float32)
+                annotation_gr = tf.cast(annotation_gr, tf.float32)
+                '''train step'''
+                loss_total, loss_low, loss_high = self.train_step(
+                    epoch=epoch, step=batch_index,
+                    total_steps=step_per_epoch,
+                    images=images,
+                    model=model,
+                    annotation_gr=annotation_gr,
+                    phi=phi,
+                    lambda_weight=self.lambda_weight,
+                    optimizer=optimizer,
+                    summary_writer=summary_writer, c_loss=c_loss)
+
+                with summary_writer.as_default():
+                    tf.summary.scalar('loss_total', loss_total, step=epoch)
+                    tf.summary.scalar('loss_low', loss_low, step=epoch)
+                    tf.summary.scalar('loss_high', loss_high, step=epoch)
+            '''save weights'''
+            model.save(self.save_path + str(epoch) + '_' + self.dataset_name + '.h5')
+
+    # @tf.function
+    def train_step(self, epoch, step, total_steps, images, model, annotation_gr, phi,
+                   optimizer, summary_writer, c_loss, lambda_weight):
+        with tf.GradientTape() as tape:
+            '''create annotation_predicted'''
+            annotation_predicted = model(images, training=True)
+            '''calculate loss'''
+            loss_total, loss_low, loss_high = c_loss.acr_loss(x_pr=annotation_predicted,
+                                                              x_gt=annotation_gr,
+                                                              phi=phi,
+                                                              lambda_weight=lambda_weight,
+                                                              ds_name=self.dataset_name)
+        '''calculate gradient'''
+        gradients_of_model = tape.gradient(loss_total, model.trainable_variables)
+        '''apply Gradients:'''
+        optimizer.apply_gradients(zip(gradients_of_model, model.trainable_variables))
+        '''printing loss Values: '''
+        tf.print("->EPOCH: ", str(epoch), "->STEP: ", str(step) + '/' + str(total_steps),
+                 ' -> : LOSS: ', loss_total,
+                 ' -> : loss_low: ', loss_low,
+                 ' -> : loss_high: ', loss_high
+                 )
+        # print('==--==--==--==--==--==--==--==--==--')
+        with summary_writer.as_default():
+            tf.summary.scalar('loss_total', loss_total, step=epoch)
+            tf.summary.scalar('loss_low', loss_low, step=epoch)
+            tf.summary.scalar('loss_high', loss_high, step=epoch)
+        return loss_total, loss_low, loss_high
+
+    def calculate_adoptive_weight(self, epoch, batch_index, y_train_filenames, weight_path):
+
+        dt_utils = DataUtil(self.num_landmark)
+        batch_y = y_train_filenames[
+                  batch_index * LearningConfig.batch_size:(batch_index + 1) * LearningConfig.batch_size]
+        asm_acc = None
+        if 0 <= epoch <= 15:
+            asm_acc = 80
+        elif 15 < epoch <= 30:
+            asm_acc = 85
+        elif 30 < epoch <= 70:
+            asm_acc = 90
+        elif 70 < epoch <= 100:
+            asm_acc = 95
+
+        pn_batch = np.array([self._load_and_normalize(self.annotation_path + file_name) for file_name in batch_y])
+        pn_batch_asm = np.array([dt_utils.get_asm(input=self._load_and_normalize(self.annotation_path + file_name),
+                                                  dataset_name=self.dataset_name, accuracy=asm_acc)
+                                 for file_name in batch_y])
+
+        delta = np.array(abs(pn_batch - pn_batch_asm))
+
+        phi = np.array([delta[i] / np.max(delta[i]) for i in range(len(pn_batch))])  # bs * num_lnd
+        return phi
+
+    def _get_optimizer(self, lr=1e-1, beta_1=0.9, beta_2=0.999, decay=1e-5):
+        return tf.keras.optimizers.Adam(lr=lr, beta_1=beta_1, beta_2=beta_2, decay=decay)
+
+    def make_model(self, arch, w_path):
+        cnn = CNNModel()
+        model = cnn.get_model(arch=arch, output_len=self.num_landmark)
+        if w_path is not None:
+            model.load_weights(w_path)
+        return model
+
+    def _shuffle_data(self, filenames, labels):
+        filenames_shuffled, y_labels_shuffled = shuffle(filenames, labels)
+        return filenames_shuffled, y_labels_shuffled
+
+    def _create_generators(self, img_path=None, annotation_path=None):
+        tf_utils = DataUtil(number_of_landmark=self.num_landmark)
+        if img_path is None:
+            filenames, labels = tf_utils.create_image_and_labels_name(img_path=self.img_path,
+                                                                      annotation_path=self.annotation_path)
+        else:
+            filenames, labels = tf_utils.create_image_and_labels_name(img_path=img_path,
+                                                                      annotation_path=annotation_path)
+        filenames_shuffled, y_labels_shuffled = shuffle(filenames, labels)
+        return filenames_shuffled, y_labels_shuffled
+
+    def _get_batch_sample(self, batch_index, x_train_filenames, y_train_filenames, is_eval=False, batch_size=None):
+        if is_eval:
+            batch_x = x_train_filenames[
+                      batch_index * batch_size:(batch_index + 1) * batch_size]
+            batch_y = y_train_filenames[
+                      batch_index * batch_size:(batch_index + 1) * batch_size]
+
+            img_batch = np.array([imread(self.eval_img_path + file_name) for file_name in batch_x]) / 255.0
+            pn_batch = np.array([load(self.eval_annotation_path + file_name) for file_name in
+                                 batch_y])
+        else:
+            img_path = self.img_path
+            pn_tr_path = self.annotation_path
+            '''create batch data and normalize images'''
+            batch_x = x_train_filenames[
+                      batch_index * LearningConfig.batch_size:(batch_index + 1) * LearningConfig.batch_size]
+            batch_y = y_train_filenames[
+                      batch_index * LearningConfig.batch_size:(batch_index + 1) * LearningConfig.batch_size]
+            '''create img and annotations'''
+            img_batch = np.array([imread(img_path + file_name) for file_name in batch_x]) / 255.0
+            pn_batch = np.array([self._load_and_normalize(pn_tr_path + file_name) for file_name in batch_y])
+
+        return img_batch, pn_batch
+
+    def _load_and_normalize(self, point_path):
+        annotation = load(point_path)
+        width = InputDataSize.image_input_size
+        height = InputDataSize.image_input_size
+        x_center = width / 2
+        y_center = height / 2
+        annotation_norm = []
+        for p in range(0, len(annotation), 2):
+            annotation_norm.append((x_center - annotation[p]) / width)
+            annotation_norm.append((y_center - annotation[p + 1]) / height)
+        return annotation_norm