Repository: openvinotoolkit/open_model_zoo (language: python)

Issue:
#dldt i was trying to convert tensorflow  fastercnn frozengraph.pb which i trained with custom data to modeloptimizer ir file i am getting this error
#dldt The Preprocessor block has been removed. Only nodes performing mean value subtraction and scaling (if applicable) are kept.
[ ERROR ] Found the following nodes '[]' with name 'swapped_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalAndROIPoolingReplacement replacement didn't work.





Discussion / hints:
@manoj652 Looks like the sub-graph replacement configuration file that you specified with the `--tensorflow_object_detection_api_pipeline_config` command line parameter is not suitable for your version of the FasterRCNN topology. Could you re-run the Model Optimizer with the `--log_level DEBUG` command line option and attach the log file?
#dldt @lazarevevgeny 

```
--log_level DEBUG
SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/ChangeCoordinateFrame/truediv_1/mul_
SecondStagePostprocessor/scale_logits/reciprocal_
SecondStagePostprocessor/scale_logits/mul_
[ ERROR ] Found the following nodes '[]' with name 'swapped_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalAndROIPoolingReplacement replacement didn't work.
[ 2018-11-15 21:55:55,244 ] [ DEBUG ] [ main:327 ] Traceback (most recent call last):
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/mo/main.py", line 321, in main
return driver(argv)
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/mo/main.py", line 263, in driver
mean_scale_values=mean_scale)
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 171, in tf2nx
class_registration.apply_replacements(graph, class_registration.ClassType.FRONT_REPLACER)
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/mo/utils/class_registration.py", line 102, in apply_replacements
replacer.find_and_replace_pattern(graph)
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/mo/front/tf/replacement.py", line 91, in find_and_replace_pattern
self.replace_sub_graph(graph, match)
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/mo/front/common/replacement.py", line 115, in replace_sub_graph
new_sub_graph = self.generate_sub_graph(graph, match)
File "/home/manojvasi/intel/computer_vision_sdk_2018.3.343/deployment_tools/model_optimizer/extensions/front/tf/ObjectDetectionAPI.py", line 261, in generate_sub_graph
format(proposal_nodes_ids))
mo.utils.error.Error: Found the following nodes '[]' with name 'swapped_proposals' but there should be exactly 1. Looks like ObjectDetectionAPIProposalAndROIPoolingReplacement replacement didn't work.
```
Is this still relevant or we can close?
Please, reopen, if necessary.

Fix patch:
diff --git a/demos/README.md b/demos/README.md
index 7da1f833ea7..84f1f3e7852 100644
--- a/demos/README.md
+++ b/demos/README.md
@@ -64,6 +64,7 @@ The Open Model Zoo includes the following demos:
 - [Single Human Pose Estimation Python\* Demo](./single_human_pose_estimation_demo/python/README.md) - 2D human pose estimation demo.
 - [Smart Classroom C++ Demo](./smart_classroom_demo/cpp/README.md) - Face recognition and action detection demo for classroom environment.
 - [Smart Classroom C++ G-API Demo](./smart_classroom_demo/cpp_gapi/README.md) - Face recognition and action detection demo for classroom environment. G-PI version.
+- [Smartlab Python\* Demo](./smartlab_demo/python/README.md) - action recognition and object detection for smartlab.
 - [Social Distance C++ Demo](./social_distance_demo/cpp/README.md) - This demo showcases a retail social distance application that detects people and measures the distance between them.
 - [Sound Classification Python\* Demo](./sound_classification_demo/python/README.md) - Demo application for sound classification algorithm.
 - [Text Detection C++ Demo](./text_detection_demo/cpp/README.md) - Text Detection demo. It detects and recognizes multi-oriented scene text on an input image and puts a bounding box around detected area.
diff --git a/demos/smartlab_demo/python/README.md b/demos/smartlab_demo/python/README.md
new file mode 100644
index 00000000000..966ed468627
--- /dev/null
+++ b/demos/smartlab_demo/python/README.md
@@ -0,0 +1,73 @@
+# Smartlab Python\* Demo
+
+This is the demo application with smartlab action recognition and smartlab object detection algorithms.
+This demo takes multi-view video inputs to identify actions and objects, then evaluates scores of current state.
+Action recognition architecure uses two encoders for front-view and top-view respectively, and a single decoder.
+Object detection uses two models for each view to detect large object and small objects, respectively.
+The following pre-trained models are delivered with the product:
+
+* `i3d-rgb-tf` + `smartlab-sequence-modelling-0001`, which are other models for identifying actions 2 actions of smartlab (adjust_rider, put_take).
+
+* `smartlab-object-detection-0001` + `smartlab-object-detection-0002` + `smartlab-object-detection-0003` + `smartlab-object-detection-0004`, which are models for detecting smartlab objects (10 objects). They detect balance, weights, tweezers, box, battery, tray, ruler, rider, scale, hand.
+
+## How It works
+
+The demo pipeline consists of several steps:
+
+* `Decode` reads frames from the input videos
+* `Detector` detects smartlab objects (balance, weights, tweezers, box, battery, tray, ruler, rider scale, hand)
+* `Segmentor` segments video frames based on action of the frame
+* `Evaluator` calculates scores of the current state
+* `Display` displays detected objects, recognized action, calculated scores on the current frame
+
+
+> **NOTE**: By default, Open Model Zoo demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the demo application or reconvert your model using the Model Optimizer tool with the `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model Using General Conversion Parameters](https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model.html#general-conversion-parameters).
+
+## Preparing to Run
+For demo input image or video files, you need to provide smartlab videos (https://storage.openvinotoolkit.org/data/test_data/videos/smartlab/).
+The list of models supported by the demo is in `<omz_dir>/demos/smartlab_demo/python/models.lst` file.
+This file can be used as a parameter for [Model Downloader](../../../tools/model_tools/README.md) and Converter to download and, if necessary, convert models to OpenVINO format (\*.xml + \*.bin).
+
+An example of using the Model Downloader:
+
+```sh
+omz_downloader --list models.lst
+```
+
+### Supported Models
+* smartlab-object-detection-0001
+* smartlab-object-detection-0002
+* smartlab-object-detection-0003
+* smartlab-object-detection-0004
+* smartlab-sequence-modelling-0001
+* i3d-rgb-tf
+
+> **NOTE**: Refer to the tables [Intel's Pre-Trained Models Device Support](../../../models/intel/device_support.md) and [Public Pre-Trained Models Device Support](../../../models/public/device_support.md) for the details on models inference support at different devices.
+
+## Running
+
+Running the application with an empty list of options yields the usage message given above and an error message.
+
+**For example**, to run the demo, please provide the model paths and two input streams:
+
+```sh
+python3 smartlab_demo.py
+    -tv ./stream_1_top.mp4
+    -fv ./stream_1_high.mp4
+    -m_ta "./intel/smartlab-object-detection-0001/FP32/smartlab-object-detection-0001.xml"
+    -m_tm "./intel/smartlab-object-detection-0002/FP32/smartlab-object-detection-0002.xml"
+    -m_fa "./intel/smartlab-object-detection-0003/FP32/smartlab-object-detection-0003.xml"
+    -m_fm "./intel/smartlab-object-detection-0004/FP32/smartlab-object-detection-0004.xml"
+    -m_en "./public/i3d-rgb-tf/FP32/i3d-rgb-tf.xml"
+    -m_de "./intel/smartlab-sequence-modelling-0001/FP32/smartlab-sequence-modelling-0001.xml"
+```
+
+## Demo Output
+
+The application uses OpenCV to display the real-time object detection, action recognition results and evaluation results.
+
+## See Also
+
+* [Open Model Zoo Demos](../../README.md)
+* [Model Optimizer](https://docs.openvinotoolkit.org/latest/_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide.html)
+* [Model Downloader](../../../tools/model_tools/README.md)
diff --git a/demos/smartlab_demo/python/display.py b/demos/smartlab_demo/python/display.py
new file mode 100644
index 00000000000..7d7318441bb
--- /dev/null
+++ b/demos/smartlab_demo/python/display.py
@@ -0,0 +1,96 @@
+"""
+ Copyright (C) 2021-2022 Intel Corporation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+"""
+
+import numpy as np
+import cv2
+
+
+class Display:
+    def __init__(self):
+        '''Score Evaluation Variables'''
+
+    def display_result(self, frame_top, frame_front, front_seg_results, top_seg_results, top_det_results, front_det_results, scoring, state, frame_counter, fps):
+        #renew score board so that when put cv2.puttext text will not overlap
+        self.score_board = np.zeros([200, 1920, 3], dtype=np.uint8)
+
+        #add action name of each frame at middle top
+        cv2.putText(frame_top, front_seg_results, (700, 80), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+        cv2.putText(frame_front, top_seg_results, (700, 80), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+
+        #display frame_number at top left corner
+        cv2.putText(frame_front, f"frame{frame_counter: 6d}", (50, 80), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+
+        #display FPS at top left corner
+        cv2.putText(frame_top, f"FPS: {fps: .2f}", (50, 160), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+        cv2.putText(frame_front, f"FPS: {fps: .2f}", (50, 160), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+
+        # show current state for troubleshooting purpose
+        cv2.putText(frame_top, state, (1500, 80), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+        cv2.putText(frame_front, state, (1500, 80), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=1.5, thickness=3)
+
+        #display obj detection result for both view
+        for row, obj_cls in zip(top_det_results[0], top_det_results[1]):
+            x_min = int(row[0])
+            y_min = int(row[1])
+            x_max = int(row[2])
+            y_max = int(row[3])
+
+            cv2.putText(frame_top, obj_cls, (x_min, y_min-10), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=0.9, thickness=2)
+            frame_top = cv2.rectangle(frame_top, (x_min, y_min), (x_max, y_max), color=(255, 0, 0), thickness=2)
+
+        for row, obj_cls in zip(front_det_results[0], front_det_results[1]):
+            x_min = int(row[0])
+            y_min = int(row[1])
+            x_max = int(row[2])
+            y_max = int(row[3])
+
+            cv2.putText(frame_front, obj_cls, (x_min, y_min-10), cv2.FONT_HERSHEY_SIMPLEX, color=(0, 0, 255), fontScale=0.9, thickness=2)
+            frame_front = cv2.rectangle(frame_front, (x_min, y_min), (x_max, y_max), color=(255, 0, 0), thickness=2)
+
+        # display scoring
+        i_rider = scoring['initial_score_rider']
+        i_balance = scoring['initial_score_balance']
+        m_rider_tweezers = scoring['measuring_score_rider_tweezers']
+        m_balance = scoring['measuring_score_balance']
+        m_object_left = scoring['measuring_score_object_left']
+        m_weights_right_tweezers = scoring['measuring_score_weights_right_tweezers']
+        m_weights_order = scoring['measuring_score_weights_order']
+        e_tidy = scoring['end_score_tidy']
+
+        cv2.putText(self.score_board,
+            f"Initial : rider[{i_rider}] balance[{i_balance}]",
+            (30, 50), cv2.FONT_HERSHEY_SIMPLEX, color=(255, 255, 255),
+            fontScale=0.9, thickness=2)
+
+        cv2.putText(self.score_board,
+            f"Measuring : object[{m_object_left}] weights_t[{m_weights_right_tweezers}] \
+            order[{m_weights_order}] rider_t[{m_rider_tweezers}] balance[{m_balance}]",
+            (30, 100), cv2.FONT_HERSHEY_SIMPLEX, color=(255, 255, 255),
+            fontScale=0.9, thickness=2)
+
+        cv2.putText(self.score_board, f"End : tidy[{e_tidy}]",
+            (30, 150), cv2.FONT_HERSHEY_SIMPLEX, color=(255, 255, 255),
+            fontScale=0.9, thickness=2)
+
+        # resize images and display them side by side, then concatenate with a scoring board to display marks
+        frame_top = cv2.resize(
+            frame_top, (int(frame_top.shape[1]/2), int(frame_top.shape[0]/2)))
+        frame_front = cv2.resize(
+            frame_front, (int(frame_front.shape[1]/2), int(frame_front.shape[0]/2)))
+
+        result_image = np.concatenate((frame_top, frame_front), axis=1)
+        result_image = np.concatenate((result_image, self.score_board), axis=0)
+        cv2.imshow("Smart Science Lab", result_image)
diff --git a/demos/smartlab_demo/python/evaluator.py b/demos/smartlab_demo/python/evaluator.py
new file mode 100644
index 00000000000..16b8632e650
--- /dev/null
+++ b/demos/smartlab_demo/python/evaluator.py
@@ -0,0 +1,226 @@
+"""
+ Copyright (C) 2021-2022 Intel Corporation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+"""
+
+import numpy as np
+
+
+class Evaluator:
+    def __init__(self):
+        '''Score Evaluation Variables'''
+        self.first_put_take = False
+        self.state = "Initial"
+        self.use_tweezers_threshold = 50    # if tweezer and rider/weight distance more than tweezer treshold, consider use hand instead of use tweezer
+        self.object_put = False # if battery is put on tray no matter left or right, return True
+        self.weights_put = False    # if weight is put on tray no matter left or right, return True
+        self.can_tidy = False # if battery and weight has been put on the tray(no matter left/right), then can tidy becomes true
+
+        # scoring
+        self.scoring = {
+            "initial_score_rider": 0,
+            "initial_score_balance": 0,
+            "measuring_score_rider_tweezers": 0,
+            "measuring_score_balance": 0,
+            "measuring_score_object_left": 0,
+            "measuring_score_weights_right_tweezers": 0,
+            "measuring_score_weights_order": 0,
+            "end_score_tidy": 0
+        }
+
+    def inference(self, top_det_results, front_det_results, top_seg_results, front_seg_results, frame_top, frame_front):
+        """
+        Args:
+            top_det_results:
+            front_det_results:
+            top_seg_results:
+            front_seg_results:
+
+        Returns:
+            Progress of the frame index
+        """
+        self.classify_state(top_seg_results)
+        if self.state == "Initial":
+            self.evaluate_rider(front_det_results)
+
+        elif self.state == "Measuring":
+            if top_seg_results == "put_take":
+                self.evaluate_object_left(front_det_results)
+                self.evaluate_weights_right_tweezers(front_det_results)
+                if not self.can_tidy:
+                    self.can_tidy = self.evaluate_end_state()
+
+            if top_seg_results == "adjust_rider":
+                self.evaluate_rider_tweezers(front_det_results)
+
+        if self.can_tidy:
+            self.evaluate_end_tidy()
+
+        return self.state, self.scoring
+
+    def classify_state(self, top_seg_results):
+        if top_seg_results == "put_take":
+            self.first_put_take = True
+
+        if self.first_put_take:
+            self.state = "Measuring"
+        else:
+            self.state = "Initial"
+
+    def evaluate_rider(self, front_det_results):
+        """
+        Function:
+            To evaluate whether rider is pushed to zero position
+
+        Args:
+            front_seg_results: action segmentation result of front view, eg: "noise_action" (type:string)
+            df: front view object detection result a single frame (type: pandas dataframe)
+        """
+        # only evaluate rider zero if 2 roundscrew and 1 rider are found
+        bboxes = front_det_results[0] # (x1, y1, x2, y2)
+        cls_ids = front_det_results[1]
+
+        if cls_ids.count("roundscrew1") == 2 and cls_ids.count("rider") == 1:
+            df_2screw = bboxes[np.array(cls_ids) == "roundscrew1"].squeeze()
+            df_rider = bboxes[np.array(cls_ids) == "rider"].squeeze()
+
+            x0 = (df_2screw[0, 1] + df_2screw[0, 0]) / 2
+            x1 = (df_2screw[1, 1] + df_2screw[1, 0]) / 2
+
+            rider_center_coordinate = \
+                ((df_rider[1] + df_rider[0]) / 2,
+                (df_rider[3] + df_rider[2]) / 2)
+
+            # if rider center position < 1/10 of length between 2 roundscrew, consider rider is pushed to zero position
+            if rider_center_coordinate[0] < min(x0, x1) + abs(x0 - x1) / 10:
+                self.scoring["initial_score_rider"] = 1
+            else:
+                self.scoring["initial_score_rider"] = 0
+
+
+    def evaluate_rider_tweezers(self, front_det_results):
+        """
+        Function:
+            To evaluate whether rider is pushed using tweezers
+
+        Args:
+            front_seg_results: action segmentation result of front view, eg: "noise_action" (type:string)
+            df: front view object detection result a single frame (type: pandas dataframe)
+        """
+
+        # only evaluate rider zero if 2 roundscrew and 1 rider are found
+        bboxes = front_det_results[0] # (x1, y1, x2, y2)
+        cls_ids = front_det_results[1]
+
+        if cls_ids.count("rider") == 1 and cls_ids.count("tweezers") == 1:
+            df_rider = bboxes[np.array(cls_ids) == "rider"].squeeze()
+            df_tweezers = bboxes[np.array(cls_ids) == "balance"].squeeze()
+            rider_min_coordinate = (df_rider[0], df_rider[2])
+            tweezers_min_coordinate = (df_tweezers[0], df_tweezers[2])
+
+            # if rider center position < 1/10 of length between 2 roundscrew, consider rider is pushed to zero position
+            if abs(rider_min_coordinate-tweezers_min_coordinate) > self.use_tweezers_threshold:
+                self.scoring['measuring_score_rider_tweezers'] = 0
+            else:
+                self.scoring['measuring_score_rider_tweezers'] = 1
+
+    def evaluate_object_left(self, front_det_results):
+        # boxes: df[1], labels: df[2], scores: df[3]
+        # [top_bboxes, top_cls_ids, top_scores],
+        bboxes = front_det_results[0] # (x1, y1, x2, y2)
+        cls_ids = front_det_results[1]
+
+        # isbattety and isbalance labels?
+        if cls_ids.count("balance") == 1 \
+            and cls_ids.count("battery") == 1:
+            df_battery = bboxes[np.array(cls_ids) == "battery"].squeeze()
+            df_balance = bboxes[np.array(cls_ids) == "balance"].squeeze()
+
+            battery_center_coordinate = \
+                ((df_battery[1] + df_battery[0]) / 2,
+                (df_battery[3] + df_battery[2]) / 2)
+            (balance_x_min, balance_x_max, balance_y_min, balance_y_max) = \
+                (df_balance[0], df_balance[1],
+                df_balance[2], df_balance[3])
+
+            if battery_center_coordinate[0] < (balance_x_min + (balance_x_max-balance_x_min) / 2) \
+                and battery_center_coordinate[0] > balance_x_min:
+
+                if battery_center_coordinate[1] > balance_y_min \
+                    and battery_center_coordinate[1] < balance_y_max:
+                    self.scoring['measuring_score_object_left'] = 1
+                    self.put_object = 1
+
+            if battery_center_coordinate[0] < balance_x_min \
+                and battery_center_coordinate[0] > balance_x_min:
+                if battery_center_coordinate[1] > balance_y_min \
+                    and battery_center_coordinate[1]<  balance_y_max:
+                    self.put_object = 1
+            else:
+                self.scoring['measuring_score_object_left'] = 0
+
+    def evaluate_weights_right_tweezers(self, front_det_results):
+        bboxes = front_det_results[0] # (x1, y1, x2, y2)
+        cls_ids = front_det_results[1]
+        if cls_ids.count("balance") == 1 and "weights" in cls_ids:
+            df_weights = bboxes[np.array(cls_ids) == "weights"].squeeze()
+            df_balance = bboxes[np.array(cls_ids) == "balance"].squeeze()
+
+            if df_weights.ndim > 1: # get only first weight box
+                df_weights = df_weights[0]
+
+            weights_center_coordinate = \
+                ((df_weights[1] + df_weights[0])/2,
+                (df_weights[3] + df_weights[2])/2)
+
+            (balance_x_min, balance_x_max, balance_y_min, balance_y_max) = \
+                (df_balance[0], df_balance[1],
+                df_balance[2], df_balance[3])
+
+            if weights_center_coordinate[0] > (balance_x_min + (balance_x_max-balance_x_min) / 2) \
+                and weights_center_coordinate[0] < balance_x_max:
+                if weights_center_coordinate[1] > balance_y_min \
+                    and weights_center_coordinate[1] < balance_y_max:
+                    self.scoring['measuring_score_weights_right_tweezers'] = 1
+                    self.weights_put = 1
+
+            if weights_center_coordinate[0] > (balance_x_min + (balance_x_max - balance_x_min) / 2) \
+                and weights_center_coordinate[0] < balance_x_max:
+                if weights_center_coordinate[1] > balance_y_min \
+                    and weights_center_coordinate[1] < balance_y_max:
+                    self.weights_put = 1
+            else:
+                self.scoring['measuring_score_weights_right_tweezers'] = 0
+
+    def evaluate_end_state(self):
+        if self.object_put == 1 and self.weights_put == 1:
+            self.can_tidy = True
+
+    def evaluate_end_tidy(self, front_det_results):
+        # define as tidied if no weights is detected(box closed),battery is on the table instead of the balance
+        bboxes = front_det_results[0] # (x1, y1, x2, y2)
+        cls_ids = front_det_results[1]
+
+        if cls_ids.count("weights") == 0:
+            if cls_ids.count("balance") == 1 and cls_ids.count("battery") == 1:
+                df_battery = bboxes[np.array(cls_ids) == "battery"].squeeze()
+                df_balance = bboxes[np.array(cls_ids) == "balance"].squeeze()
+
+                battery_center_coordinate = \
+                    ((df_battery[1] + df_battery[0]) / 2,
+                    (df_battery[3] + df_battery[2]) / 2)
+                balance_y_max = df_balance[3]
+
+                if battery_center_coordinate[1] < balance_y_max:
+                    self.scoring["end_score_tidy"] = 1
diff --git a/demos/smartlab_demo/python/init_his.npz b/demos/smartlab_demo/python/init_his.npz
new file mode 100644
index 00000000000..ac75b816ba2
Binary files /dev/null and b/demos/smartlab_demo/python/init_his.npz differ
diff --git a/demos/smartlab_demo/python/models.lst b/demos/smartlab_demo/python/models.lst
new file mode 100644
index 00000000000..3b6aaebf2be
--- /dev/null
+++ b/demos/smartlab_demo/python/models.lst
@@ -0,0 +1,7 @@
+# This file can be used with the --list option of the model downloader.
+smartlab-object-detection-0001
+smartlab-object-detection-0002
+smartlab-object-detection-0003
+smartlab-object-detection-0004
+smartlab-sequence-modelling-0001
+i3d-rgb-tf
diff --git a/demos/smartlab_demo/python/object_detection/__init__.py b/demos/smartlab_demo/python/object_detection/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/demos/smartlab_demo/python/object_detection/deploy_util.py b/demos/smartlab_demo/python/object_detection/deploy_util.py
new file mode 100644
index 00000000000..bffd8da93c7
--- /dev/null
+++ b/demos/smartlab_demo/python/object_detection/deploy_util.py
@@ -0,0 +1,79 @@
+# Copyright (c) 2014-2021 Megvii Inc. All rights reserved.
+
+import numpy as np
+
+def nms(boxes, scores, nms_thr):
+    """Single class NMS implemented in Numpy."""
+    x1 = boxes[:, 0]
+    y1 = boxes[:, 1]
+    x2 = boxes[:, 2]
+    y2 = boxes[:, 3]
+
+    areas = (x2 - x1 + 1) * (y2 - y1 + 1)
+    order = scores.argsort()[::-1]
+
+    keep = []
+    while order.size > 0:
+        i = order[0]
+        keep.append(i)
+        xx1 = np.maximum(x1[i], x1[order[1:]])
+        yy1 = np.maximum(y1[i], y1[order[1:]])
+        xx2 = np.minimum(x2[i], x2[order[1:]])
+        yy2 = np.minimum(y2[i], y2[order[1:]])
+
+        w = np.maximum(0.0, xx2 - xx1 + 1)
+        h = np.maximum(0.0, yy2 - yy1 + 1)
+        inter = w * h
+        ovr = inter / (areas[i] + areas[order[1:]] - inter)
+
+        inds = np.where(ovr <= nms_thr)[0]
+        order = order[inds + 1]
+
+    return keep
+
+def multiclass_nms(boxes, scores, nms_thr, score_thr):
+    """Multiclass NMS implemented in Numpy"""
+    final_dets = []
+    num_classes = scores.shape[1]
+    for cls_ind in range(num_classes):
+        cls_scores = scores[:, cls_ind]
+        valid_score_mask = cls_scores > score_thr
+        if valid_score_mask.sum() == 0:
+            continue
+        else:
+            valid_scores = cls_scores[valid_score_mask]
+            valid_boxes = boxes[valid_score_mask]
+            keep = nms(valid_boxes, valid_scores, nms_thr)
+            if len(keep) > 0:
+                cls_inds = np.ones((len(keep), 1)) * cls_ind
+                dets = np.concatenate([valid_boxes[keep], valid_scores[keep, None], cls_inds], 1)
+                final_dets.append(dets)
+    if len(final_dets) == 0:
+        return None
+    return np.concatenate(final_dets, 0)
+
+def demo_postprocess(outputs, img_size, p6=False):
+    grids = []
+    expanded_strides = []
+
+    if not p6:
+        strides = [8, 16, 32]
+    else:
+        strides = [8, 16, 32, 64]
+
+    hsizes = [img_size[0] // stride for stride in strides]
+    wsizes = [img_size[1] // stride for stride in strides]
+
+    for hsize, wsize, stride in zip(hsizes, wsizes, strides):
+        xv, yv = np.meshgrid(np.arange(wsize), np.arange(hsize))
+        grid = np.stack((xv, yv), 2).reshape(1, -1, 2)
+        grids.append(grid)
+        shape = grid.shape[:2]
+        expanded_strides.append(np.full((*shape, 1), stride))
+
+    grids = np.concatenate(grids, 1)
+    expanded_strides = np.concatenate(expanded_strides, 1)
+    outputs[..., :2] = (outputs[..., :2] + grids) * expanded_strides
+    outputs[..., 2:4] = np.exp(outputs[..., 2:4]) * expanded_strides
+
+    return outputs
diff --git a/demos/smartlab_demo/python/object_detection/detector.py b/demos/smartlab_demo/python/object_detection/detector.py
new file mode 100644
index 00000000000..48544d8c751
--- /dev/null
+++ b/demos/smartlab_demo/python/object_detection/detector.py
@@ -0,0 +1,214 @@
+"""
+ Copyright (C) 2021-2022 Intel Corporation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+"""
+
+import numpy as np
+from .preprocess import preprocess
+from .settings import MwGlobalExp
+from .deploy_util import multiclass_nms, demo_postprocess
+from .subdetectors import SubDetector
+
+
+class Detector:
+    def __init__(self,
+        ie,
+        device,
+        fp_top_models: list,
+        fp_front_models: list,
+        backend: str='openvino'):
+
+        '''Object Detection Variables'''
+        self.object_detection_boxes = []
+        self.front_anno_text_files = None
+        self.top_anno_text_files = None
+
+        '''configure output/preview'''
+        self.save_result = False
+        self.backend = backend
+
+        '''configure settings for 2 models in top view'''
+        ###          topview.global_subdetector1           ###
+        #  max-number constraints:
+        #     "balance", 1; "weights",    omit; "tweezers", 1;
+        #     "box"    , 1; "battery", 1; "tray"    , 2;
+        #     "ruler"  , 1; "rider"  ,    omit; "scale"   , 1;
+        #     "hand", 2;
+        #  other conditions:
+        #     conf 0.1; nms 0.3
+        self.top1_exp = MwGlobalExp(
+            num_classes = 10,
+            fp_model = fp_top_models[0],
+            conf_thresh= 0.1,
+            nms_thresh = 0.3,
+            ie=ie,
+            device=device)
+
+        ###           topview.global_subdetector2          ###
+        #  max-number constraints:
+        #     "weights", 6; "tweezers", 1; "battery", 1;
+        #  other conditions:
+        #     conf 0.1; nms 0.2
+        self.top2_exp = MwGlobalExp(
+            num_classes = 3,
+            fp_model = fp_top_models[1],
+            conf_thresh= 0.1,
+            nms_thresh = 0.2,
+            ie=ie,
+            device=device)
+
+        '''configure settings for 2 models in front view'''
+        ###          frontview.global_subdetector1           ###
+        #  max-number constraints:
+        #     "balance", 1; "weights",    omit; "tweezers", 1;
+        #     "box"    , 1; "battery", 1; "tray"    , 2;
+        #     "ruler"  , 1; "rider"  ,    omit; "scale"   , 1;
+        #     "hand", 2;
+        #  other conditions:
+        #     conf 0.2; nms 0.3
+        self.front1_exp = MwGlobalExp(
+            num_classes = 10,
+            fp_model = fp_front_models[0],
+            conf_thresh= 0.2,
+            nms_thresh = 0.3,
+            ie=ie,
+            device=device)
+
+        ###           frontview.global_subdetector2          ###
+        #  max-number constraints:
+        #     "weights", 6; "tweezers", 1; "battery", 1;
+        #  other conditions:
+        #     conf 0.1; nms 0.2
+        self.front2_exp = MwGlobalExp(
+            num_classes = 3,
+            fp_model = fp_front_models[1],
+            conf_thresh= 0.1,
+            nms_thresh = 0.2,
+            ie=ie,
+            device=device)
+
+        ### concatenate list of class names for topview
+        cls1 = self.top1_exp.mw_classes
+        cls2 = self.top2_exp.mw_classes
+        self.classes = cls1 + cls2
+        self.offset_cls_idx = [0, len(self.top1_exp.mw_classes)]
+        #  max-number constraints:
+        self.max_nums = {
+            "balance": 1, "weights": 6, "tweezers": 1,
+            "box"    : 1, "battery": 1, "tray"    : 2,
+            "ruler"  : 1, "rider"  : 1, "scale"   : 1,
+            "hand": 2,
+        }
+
+        ### build map of overlap classes
+        self.cls2tocls1 = {}
+        cls1_c2i = {c: i for i, c in enumerate(cls1)}
+        cls2_c2i = {c: (i + self.offset_cls_idx[1]) for i, c in enumerate(cls2)}
+        for cls in cls2:
+            if cls in cls1:
+                idx1 = cls1_c2i[cls]
+                idx2 = cls2_c2i[cls]
+                self.cls2tocls1[idx2] = idx1
+        self.repeat_cls2_ids = self.cls2tocls1.keys()
+        self.norepeat_num_classes = len(self.classes) - len(self.repeat_cls2_ids)
+
+        ### load models for top view
+        self.top1_subdetector = SubDetector(self.top1_exp, self.backend)
+        self.top2_subdetector = SubDetector(self.top2_exp, self.backend)
+        ### load models for front view
+        self.front1_subdetector = SubDetector(self.front1_exp, self.backend)
+        self.front2_subdetector = SubDetector(self.front2_exp, self.backend)
+
+    def _detect_one(self, img, view='top'):
+        if view == 'top': # top view
+            sub_detector1 = self.top1_subdetector
+            sub_detector2 = self.top2_subdetector
+        else: # front view
+            sub_detector1 = self.front1_subdetector
+            sub_detector2 = self.front2_subdetector
+
+        all_preds = []
+        for i, sub_detector in enumerate([sub_detector1, sub_detector2]):
+            outputs, img_info = sub_detector.inference(img)
+            if outputs[0] is not None:
+                preds = outputs[0] # work if bsize = 1
+            else:
+                continue
+            preds[:, 6] += self.offset_cls_idx[i]
+
+            if i == 0:
+                all_preds = np.array(preds)
+            elif len(all_preds) == 0:
+                all_preds = np.zeros((1, 7))
+            else:
+                np.vstack((all_preds, preds))
+
+        # merge same classes from model 2
+        for r, pred in enumerate(all_preds):
+            cls_id = int(pred[-1])
+            if cls_id in self.repeat_cls2_ids:
+                all_preds[r, -1] = self.cls2tocls1[cls_id]
+
+        # remap to original image scale
+        ratio = np.array(img_info['ratio'])
+        bboxes = all_preds[:, :4] / ratio
+        cls = all_preds[:, 6]
+        scores = all_preds[:, 4] * all_preds[:, 5]
+
+        return bboxes, cls, scores
+
+    def pseudo_detect(self, origin_img, input_blob, out_blob, exec_net, h, w, idx_offset: int):
+        image, ratio = preprocess(origin_img, (h, w))
+        res = exec_net.infer(inputs={input_blob: image})
+        res = res[out_blob]
+
+        predictions = demo_postprocess(res, (h, w), p6=False)[0]
+        boxes = predictions[:, :4]
+        scores = predictions[:, 4, None] * predictions[:, 5:]
+        boxes_xyxy = np.ones_like(boxes)
+        boxes_xyxy[:, 0] = boxes[:, 0] - boxes[:, 2]/2. # x0,y0,x1,y1
+        boxes_xyxy[:, 1] = boxes[:, 1] - boxes[:, 3]/2.
+        boxes_xyxy[:, 2] = boxes[:, 0] + boxes[:, 2]/2.
+        boxes_xyxy[:, 3] = boxes[:, 1] + boxes[:, 3]/2.
+        boxes_xyxy /= ratio
+        dets = multiclass_nms(boxes_xyxy, scores, nms_thr=0.45, score_thr=0.1)
+
+        if dets is not None:
+            final_boxes = dets[:, :4]
+            final_scores, final_cls_inds = dets[:, 4], dets[:, 5]
+        else:
+            return idx_offset, []
+
+        return final_boxes, final_cls_inds, final_scores
+
+
+    def inference(self, img_top, img_front):
+        """
+        todo Given input arrays for two view, need to generate and save the corresponding detection results
+            in the specific data structure.
+        Args:
+        img_top: img array of H x W x C for the top view
+        img_front: img_array of H x W x C for the front view
+
+        Returns:
+        prediction results for the two images
+        """
+        top_bboxes, top_cls_ids, top_scores = self._detect_one(img_top, view='top')
+        front_bboxes, front_cls_ids, front_scores = self._detect_one(img_front, view='front')
+
+        # get class string
+        top_cls_ids = [ self.classes[int(x)] for x in top_cls_ids ]
+        front_cls_ids = [ self.classes[int(x)] for x in front_cls_ids ]
+
+        return [top_bboxes, top_cls_ids, top_scores], [front_bboxes, front_cls_ids, front_scores]
diff --git a/demos/smartlab_demo/python/object_detection/postprocess.py b/demos/smartlab_demo/python/object_detection/postprocess.py
new file mode 100644
index 00000000000..5f98ea22a12
--- /dev/null
+++ b/demos/smartlab_demo/python/object_detection/postprocess.py
@@ -0,0 +1,52 @@
+import numpy as np
+from .deploy_util import nms, multiclass_nms
+
+
+def postprocess(prediction, num_classes, conf_thre=0.7, nms_thre=0.45, class_agnostic=False):
+    # original source: https://github.com/Megvii-BaseDetection/YOLOX
+    # Copyright (c) 2014-2021 Megvii Inc. All rights reserved.
+    box_corner = np.zeros(prediction.shape)
+
+    # box_corner = prediction.new(prediction.shape)
+    box_corner[:, :, 0] = prediction[:, :, 0] - prediction[:, :, 2] / 2
+    box_corner[:, :, 1] = prediction[:, :, 1] - prediction[:, :, 3] / 2
+    box_corner[:, :, 2] = prediction[:, :, 0] + prediction[:, :, 2] / 2
+    box_corner[:, :, 3] = prediction[:, :, 1] + prediction[:, :, 3] / 2
+    prediction[:, :, :4] = box_corner[:, :, :4]
+    output = [None for _ in range(len(prediction))]
+
+    for i, image_pred in enumerate(prediction):
+        # If none are remaining => process next image
+        if len(image_pred[0]) == 0:
+            continue
+        # Get score and class with highest confidence
+        class_conf = np.max(image_pred[:, 5:5+num_classes], axis=1, keepdims=True)
+        class_pred = np.argmax(image_pred[: , 5:5+num_classes], axis=1)
+        class_pred = np.expand_dims(class_pred, axis=1)
+
+        conf_mask = (image_pred[:, 4] * class_conf.squeeze() >= conf_thre).squeeze()
+        # Detections ordered as (x1, y1, x2, y2, obj_conf, class_conf, class_pred)
+        detections = np.concatenate((image_pred[:, :5], class_conf, class_pred), axis=1)
+        detections = detections[conf_mask]
+
+        if len(detections)<1:
+            continue
+        if class_agnostic:
+            nms_out_index = nms(
+                detections[:, :4],
+                detections[:, 4] * detections[:, 5],
+                nms_thre)
+        else:
+            nms_out_index = multiclass_nms(
+                detections[:, :4],
+                detections[:, 4] * detections[:, 5],
+                detections[:, 6],
+                nms_thre)
+
+        detections = detections[nms_out_index] # filtering boxes
+
+        if output[i] is None:
+            output[i] = detections
+        else:
+            output[i] = np.concatenate((output[i], detections))
+    return output
diff --git a/demos/smartlab_demo/python/object_detection/preprocess.py b/demos/smartlab_demo/python/object_detection/preprocess.py
new file mode 100644
index 00000000000..21015953d7e
--- /dev/null
+++ b/demos/smartlab_demo/python/object_detection/preprocess.py
@@ -0,0 +1,21 @@
+import cv2
+import numpy as np
+
+
+def preprocess(img, input_size, swap=(2, 0, 1)):
+    if len(img.shape) == 3:
+        padded_img = np.ones((input_size[0], input_size[1], 3), dtype=np.uint8) * 114
+    else:
+        padded_img = np.ones(input_size, dtype=np.uint8) * 114
+
+    r = min(input_size[0] / img.shape[0], input_size[1] / img.shape[1])
+    resized_img = cv2.resize(
+        img,
+        (int(img.shape[1] * r), int(img.shape[0] * r)),
+        interpolation=cv2.INTER_LINEAR,
+    ).astype(np.uint8)
+    padded_img[: int(img.shape[0] * r), : int(img.shape[1] * r)] = resized_img
+
+    padded_img = padded_img.transpose(swap)
+    padded_img = np.ascontiguousarray(padded_img, dtype=np.float32)
+    return padded_img, r
diff --git a/demos/smartlab_demo/python/object_detection/settings.py b/demos/smartlab_demo/python/object_detection/settings.py
new file mode 100644
index 00000000000..7b74406e816
--- /dev/null
+++ b/demos/smartlab_demo/python/object_detection/settings.py
@@ -0,0 +1,61 @@
+# Copyright (c) Megvii, Inc. and its affiliates.
+"""
+ Copyright (C) 2021-2022 Intel Corporation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+"""
+
+# class maps
+mw_glb2bcls3 = (
+  'weights',
+  'tweezers',
+  'battery'
+)
+
+mw_glb1cls10 = (
+  "balance",
+  "weights",
+  "tweezers",
+  "box",
+  "battery",
+  "tray",
+  "ruler",
+  "rider",
+  "scale",
+  "hand"
+)
+
+# global setting of obj-det
+class MwGlobalExp:
+    def __init__(self, num_classes, fp_model, nms_thresh, conf_thresh, ie, device):
+        if num_classes == 10:
+            self.mw_classes = mw_glb1cls10
+        elif num_classes == 3:
+            self.mw_classes = mw_glb2bcls3
+        else:
+            raise ValueError(f'num_classes={num_classes} is not supported, use 10 or 3')
+        # define model file
+        self.fp_model = fp_model
+        self.conf_thresh = conf_thresh
+        self.nms_thresh = nms_thresh
+        self.num_classes = num_classes
+        self.ie = ie
+        self.device = device
+
+    def get_openvino_model(self):
+        net = self.ie.read_model(self.fp_model)
+        compiled_model = self.ie.compile_model(model=net, device_name=self.device)
+        input_name = compiled_model.inputs[0]
+        output_name = compiled_model.outputs[0]
+
+        return (input_name, output_name, (416, 416), compiled_model)
diff --git a/demos/smartlab_demo/python/object_detection/subdetectors.py b/demos/smartlab_demo/python/object_detection/subdetectors.py
new file mode 100644
index 00000000000..c826cc38ce1
--- /dev/null
+++ b/demos/smartlab_demo/python/object_detection/subdetectors.py
@@ -0,0 +1,66 @@
+import numpy as np
+from .preprocess import preprocess
+from .postprocess import postprocess
+from .settings import MwGlobalExp
+from .deploy_util import demo_postprocess
+
+
+class SubDetector:
+    def __init__(self, exp: MwGlobalExp, backend: str='openvino'):
+        self.inode, self.onode, self.input_shape, self.model = exp.get_openvino_model()
+
+        self.cls_names = exp.mw_classes
+        self.num_classes = exp.num_classes
+        self.conf_thresh = exp.conf_thresh
+        self.nms_thresh = exp.nms_thresh
+        self.infer_request = self.model.create_infer_request()
+
+    def inference(self, img):
+        img_info = {"id": 0}
+        height, width = img.shape[:2]
+        img_info["height"] = height
+        img_info["width"] = width
+
+        img_feed, ratio = preprocess(img, self.input_shape)
+        img_info["ratio"] = ratio
+        res = self.infer_request.infer({self.inode: np.expand_dims(img_feed, axis=0)})[self.onode]
+        outputs = demo_postprocess(res, self.input_shape, p6=False)
+
+        outputs = postprocess(
+            outputs, self.num_classes, self.conf_thresh,
+            self.nms_thresh, class_agnostic=True)
+
+        return outputs, img_info
+
+    def pseudolabel(self, output, img_info, idx_offset, cls_conf=0.35):
+        ratio = img_info["ratio"]
+        img = img_info["raw_img"]
+        image_id = img_info['id']
+        if output is None:
+            return img
+        bboxes = output[:, 0:4]
+        # preprocessing: resize
+        bboxes /= ratio # [[x0,y0,x1,y1], ...]
+        cls = output[:, 6]
+        scores = output[:, 4] * output[:, 5]
+        # enumerate all bbox
+        i=0
+        res = []
+        for box, c, s in zip(bboxes, cls, scores):
+            if s < cls_conf:
+                continue
+            x0, y0, x1, y1 = int(box[0]), int(box[1]), int(box[2]), int(box[3])
+            w, h = x1-x0, y1-y0
+            idx = idx_offset + i
+            i += 1
+            cat_id = int(c)
+            res.append({
+                'area': w * h,
+                'bbox': [x0, y0, w, h],
+                'category_id': cat_id,
+                'id': idx,
+                'image_id': image_id,
+                'iscrowd': 0,
+                'segmentation': [[x0, y0, x1, y1]]
+            })
+        return res
diff --git a/demos/smartlab_demo/python/segmentor.py b/demos/smartlab_demo/python/segmentor.py
new file mode 100644
index 00000000000..f729addb6df
--- /dev/null
+++ b/demos/smartlab_demo/python/segmentor.py
@@ -0,0 +1,208 @@
+"""
+ Copyright (C) 2021-2022 Intel Corporation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+"""
+
+import cv2
+import numpy as np
+import logging as log
+from pathlib import Path
+from scipy.special import softmax
+from openvino.runtime import PartialShape
+
+class SegmentorMstcn:
+    def __init__(self, ie, device, i3d_path, mstcn_path):
+        self.ActionTerms = [
+            "background",
+            "noise_action",
+            "remove_support_sleeve",
+            "remove_pointer_sleeve",
+            "adjust_rider",
+            "adjust_nut",
+            "adjust_balancing",
+            "open_box",
+            "close_box",
+            "choose_weight",
+            "put_left",
+            "put_right",
+            "take_left",
+            "take_right",
+            "install_support_sleeve",
+            "install_pointer_sleeve",
+        ]
+
+        self.EmbedBufferTop = np.zeros((1024, 0))
+        self.EmbedBufferFront = np.zeros((1024, 0))
+        self.ImgSizeHeight = 224
+        self.ImgSizeWidth = 224
+        self.EmbedBatchSize = 1
+        self.SegBatchSize = 24
+        self.EmbedWindowLength = 16
+        self.EmbedWindowStride = 1
+        self.EmbedWindowAtrous = 3
+        self.TemporalLogits = np.zeros((0, len(self.ActionTerms)))
+
+        net = ie.read_model(i3d_path)
+        net.reshape({net.inputs[0]: PartialShape(
+            [self.EmbedBatchSize, self.EmbedWindowLength, self.ImgSizeHeight, self.ImgSizeWidth, 3])})
+        nodes = net.get_ops()
+        net.add_outputs(nodes[13].output(0))
+        self.i3d = ie.compile_model(model=net, device_name=device)
+
+        self.mstcn_net = ie.read_model(mstcn_path)
+        self.mstcn = ie.compile_model(model=self.mstcn_net, device_name=device)
+        self.mstcn_input_keys = self.mstcn.inputs
+        self.mstcn_output_key = self.mstcn.outputs
+        self.mstcn_net.reshape({'input': PartialShape([1, 2048, 1])})
+        self.reshape_mstcn = ie.compile_model(model=self.mstcn_net, device_name=device)
+        file_path = Path(__file__).parent / 'init_his.npz'
+        init_his_feature = np.load(file_path)
+        self.his_fea = {f'fhis_in_{i}': init_his_feature[f'arr_{i}'] for i in range(4)}
+
+    def inference(self, buffer_top, buffer_front, frame_index):
+        """
+        Args:
+            buffer_top: buffers of the input image arrays for the top view
+            buffer_front: buffers of the input image arrays for the front view
+            frame_index: frame index of the latest frame
+        Returns: the temporal prediction results for each frame (including the historical predictions)，
+                 length of predictions == frame_index()
+        """
+        ### run encoder ###
+        self.EmbedBufferTop = self.feature_embedding(
+            img_buffer=buffer_top,
+            embedding_buffer=self.EmbedBufferTop,
+            frame_index=frame_index)
+        self.EmbedBufferFront = self.feature_embedding(
+            img_buffer=buffer_front,
+            embedding_buffer=self.EmbedBufferFront,
+            frame_index=frame_index)
+
+        ### run mstcn++ only batch size 1###
+        if min(self.EmbedBufferTop.shape[-1], self.EmbedBufferFront.shape[-1]) > 0:
+            self.action_segmentation()
+
+        # ### get label ###
+        valid_index = self.TemporalLogits.shape[0]
+        if valid_index == 0:
+            return []
+        else:
+            frame_predictions = [self.ActionTerms[i] for i in np.argmax(self.TemporalLogits, axis=1)]
+            frame_predictions = ["background" for i in range(self.EmbedWindowLength - 1)] + frame_predictions
+
+        return frame_predictions[-1]
+
+    def feature_embedding(self, img_buffer, embedding_buffer, frame_index):
+        # minimal temporal length for processor
+        min_t = (self.EmbedWindowLength - 1) * self.EmbedWindowAtrous
+
+        infer_request = self.i3d.create_infer_request()
+        if frame_index > min_t:
+            num_embedding = embedding_buffer.shape[-1]
+            img_buffer = list(img_buffer)
+            curr_t = self.EmbedWindowStride * num_embedding + (self.EmbedWindowLength - 1) * self.EmbedWindowAtrous
+            while curr_t < frame_index:
+                # absolute index in temporal shaft
+                start_index = self.EmbedWindowStride * num_embedding
+
+                if frame_index > len(img_buffer):
+                    # absolute index in buffer shaft
+                    start_index = start_index - (frame_index - len(img_buffer))
+
+                input_data = [
+                    [cv2.resize(img_buffer[start_index + i * self.EmbedWindowAtrous],
+                    (self.ImgSizeHeight, self.ImgSizeWidth)) for i in range(self.EmbedWindowLength)]
+                    for j in range(self.EmbedBatchSize)]
+                input_data = np.asarray(input_data).transpose((0, 4, 1, 2, 3))
+                input_data = input_data * 127.5 + 127.5
+
+                input_dict = {self.i3d.inputs[0]: input_data}
+                out_logits = infer_request.infer(input_dict)[self.i3d.outputs[1]]
+                out_logits = out_logits.squeeze((0, 3, 4))
+
+                # ndarray: C x num_embedding
+                embedding_buffer = np.concatenate((embedding_buffer, out_logits), axis=1)
+
+                curr_t += self.EmbedWindowStride
+        return embedding_buffer
+
+    def action_segmentation(self):
+        # read buffer
+        embed_buffer_top = self.EmbedBufferTop
+        embed_buffer_front = self.EmbedBufferFront
+        batch_size = self.SegBatchSize
+        start_index = self.TemporalLogits.shape[0]
+        end_index = min(embed_buffer_top.shape[-1], embed_buffer_front.shape[-1])
+        num_batch = (end_index - start_index) // batch_size
+
+        infer_request = self.reshape_mstcn.create_infer_request()
+        if num_batch < 0:
+            log.debug("Waiting for the next frame ...")
+        elif num_batch == 0:
+            log.debug(f"start_index: {start_index} end_index: {end_index}")
+            unit1 = embed_buffer_top[:, start_index:end_index]
+            unit2 = embed_buffer_front[:, start_index:end_index]
+            feature_unit = np.concatenate([unit1[:, ], unit2[:, ]], axis=0)
+            input_mstcn = np.expand_dims(feature_unit, 0)
+
+            feed_dict = {}
+            if len(self.his_fea) != 0:
+                for key in self.mstcn_input_keys:
+                    if 'fhis_in_' in str(key.names):
+                        string = list(key.names)[0]
+                        feed_dict[string] = self.his_fea[string]
+            feed_dict['input'] = input_mstcn
+            if input_mstcn.shape == (1, 2048, 1):
+                out = infer_request.infer(feed_dict)
+
+            predictions = out[list(out.keys())[-1]]
+            for key in self.mstcn_output_key:
+                if 'fhis_in_' in str(key.names):
+                    string = list(key.names)[0]
+                    self.his_fea[string] = out[string]
+
+            """
+                predictions --> 4x1x64x24
+                his_fea --> [12*[1x64x2048], 11*[1x64x2048], 11*[1x64x2048], 11*[1x64x2048]]
+            """
+            temporal_logits = predictions[:, :, :len(self.ActionTerms), :]  # 4x1x16xN
+            temporal_logits = softmax(temporal_logits[-1], 1)  # 1x16xN
+            temporal_logits = temporal_logits.transpose((0, 2, 1)).squeeze(axis=0)
+            self.TemporalLogits = np.concatenate([self.TemporalLogits, temporal_logits], axis=0)
+        else:
+            for batch_idx in range(num_batch):
+                unit1 = embed_buffer_top[:,
+                        start_index + batch_idx * batch_size:start_index + batch_idx * batch_size + batch_size]
+                unit2 = embed_buffer_front[:,
+                        start_index + batch_idx * batch_size:start_index + batch_idx * batch_size + batch_size]
+                feature_unit = np.concatenate([unit1[:, ], unit2[:, ]], axis=0)
+
+                feed_dict = {}
+                if len(self.his_fea) != 0:
+                    for key in self.mstcn_input_keys:
+                        if 'fhis_in_' in str(key.names):
+                            string = list(key.names)[0]
+                            feed_dict[key] = self.his_fea[string]
+                feed_dict['input'] = feature_unit
+                out = infer_request.infer(feed_dict)
+                predictions = out[list(out.keys())[-1]]
+                for key in self.mstcn_output_key:
+                    if 'fhis_in_' in str(key.names):
+                        string = list(key.names)[0]
+                        self.his_fea[string] = out[string]
+
+                temporal_logits = predictions[:, :, :len(self.ActionTerms), :]  # 4x1x16xN
+                temporal_logits = softmax(temporal_logits[-1], 1)  # 1x16xN
+                temporal_logits = temporal_logits.transpose((0, 2, 1)).squeeze(axis=0)
+                self.TemporalLogits = np.concatenate([self.TemporalLogits, temporal_logits], axis=0)
diff --git a/demos/smartlab_demo/python/smartlab_demo.py b/demos/smartlab_demo/python/smartlab_demo.py
new file mode 100644
index 00000000000..b5f4df718c4
--- /dev/null
+++ b/demos/smartlab_demo/python/smartlab_demo.py
@@ -0,0 +1,144 @@
+"""
+ Copyright (C) 2021-2022 Intel Corporation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+"""
+
+import cv2
+import time
+from collections import deque
+from argparse import ArgumentParser, SUPPRESS
+from object_detection.detector import Detector
+from segmentor import SegmentorMstcn
+from evaluator import Evaluator
+from display import Display
+from openvino.runtime import Core
+
+
+def build_argparser():
+    parser = ArgumentParser(add_help=False)
+    args = parser.add_argument_group('Options')
+    args.add_argument('-h', '--help', action='help', default=SUPPRESS,
+                    help='Show this help message and exit.')
+    args.add_argument("-d", "--device", type=str, default='CPU', required=False,
+                        help="Optional. Specify the target to infer on CPU or GPU.")
+    args.add_argument('-tv', '--topview', required=True,
+                    help='Required. Topview stream to be processed. The input must be a single image, '
+                    'a folder of images, video file or camera id.')
+    args.add_argument('-fv', '--frontview', required=True,
+                    help='Required. FrontView to be processed. The input must be a single image, '
+                    'a folder of images, video file or camera id.')
+    args.add_argument('-m_ta', '--m_topall', help='Required. Path to topview all class model.', required=True, type=str)
+    args.add_argument('-m_tm', '--m_topmove', help='Required. Path to topview moving class model.', required=True, type=str)
+    args.add_argument('-m_fa', '--m_frontall', help='Required. Path to frontview all class model.', required=True, type=str)
+    args.add_argument('-m_fm', '--m_frontmove', help='Required. Path to frontview moving class model.', required=True, type=str)
+    args.add_argument('-m_en', '--m_encoder', help='Required. Path to encoder model.', required=True, type=str)
+    args.add_argument('-m_de', '--m_decoder', help='Required. Path to decoder model.', required=True, type=str)
+
+    return parser
+
+def main():
+    args = build_argparser().parse_args()
+
+    frame_counter = 0 # Frame index counter
+    buffer1 = deque(maxlen=1000)  # Array buffer
+    buffer2 = deque(maxlen=1000)
+    ie = Core()
+
+    ''' Object Detection Variables'''
+    detector = Detector(
+            ie,
+            args.device,
+            [args.m_topall, args.m_topmove],
+            [args.m_frontall, args.m_frontmove])
+
+    '''Video Segmentation Variables'''
+    segmentor = SegmentorMstcn(ie, args.device, args.m_encoder, args.m_decoder)
+
+    '''Score Evaluation Variables'''
+    evaluator = Evaluator()
+
+    '''Display Obj Detection, Action Segmentation and Score Evaluation Result'''
+    display = Display()
+
+    """
+        Process the video.
+    """
+    cap_top = cv2.VideoCapture(args.topview)
+    cap_front = cv2.VideoCapture(args.frontview)
+
+    old_time = time.time()
+    fps = 0.0
+    interval_second = 1
+    interval_start_frame = 0
+    total_frame_processed_in_interval = 0.0
+    while cap_top.isOpened() and cap_front.isOpened():
+        ret_top, frame_top = cap_top.read()  # frame:480 x 640 x 3
+        ret_front, frame_front = cap_front.read()
+
+        if not ret_top or not ret_front:
+            break
+        else:
+            frame_counter += 1
+
+            ''' The object detection module need to generate detection results(for the current frame) '''
+            top_det_results, front_det_results = detector.inference(
+                    img_top=frame_top, img_front=frame_front)
+
+            ''' The temporal segmentation module need to self judge and generate segmentation results for all historical frames '''
+            buffer1.append(cv2.cvtColor(frame_top, cv2.COLOR_BGR2RGB))
+            buffer2.append(cv2.cvtColor(frame_front, cv2.COLOR_BGR2RGB))
+
+            frame_predictions = segmentor.inference(
+                    buffer_top=buffer1,
+                    buffer_front=buffer2,
+                    frame_index=frame_counter)
+            top_seg_results = frame_predictions
+            front_seg_results = frame_predictions
+            if(len(top_seg_results) == 0):
+                continue
+
+            ''' The score evaluation module need to merge the results of the two modules and generate the scores '''
+            state, scoring = evaluator.inference(
+                    top_det_results=top_det_results,
+                    front_det_results=front_det_results,
+                    top_seg_results=top_seg_results,
+                    front_seg_results=front_seg_results,
+                    frame_top=frame_top,
+                    frame_front=frame_front)
+
+            current_time=time.time()
+            current_frame = frame_counter
+            if (current_time - old_time > interval_second):
+                total_frame_processed_in_interval = current_frame - interval_start_frame
+                fps = total_frame_processed_in_interval / (current_time - old_time)
+                interval_start_frame = current_frame
+                old_time = current_time
+
+            display.display_result(
+                    frame_top=frame_top,
+                    frame_front=frame_front,
+                    front_seg_results=front_seg_results,
+                    top_seg_results=top_seg_results,
+                    top_det_results=top_det_results,
+                    front_det_results=front_det_results,
+                    scoring=scoring,
+                    state=state,
+                    frame_counter=frame_counter,
+                    fps=fps)
+
+            if cv2.waitKey(1) in {ord('q'), ord('Q'), 27}: # Esc
+                break
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/demos/tests/cases.py b/demos/tests/cases.py
index 746c9be377a..701401fad5a 100644
--- a/demos/tests/cases.py
+++ b/demos/tests/cases.py
@@ -1240,6 +1240,21 @@ def single_option_cases(key, *args):
     #    ]
     #)),
 
+    PythonDemo(name='smartlab_demo', device_keys=['-d'],
+        model_keys=['-m_ta', '-m_tm', '-m_fa', '-m_fm', '-m_en', '-m_de'],
+        test_cases=combine_cases(
+        [
+            TestCase(options={'-tv': TestDataArg('data/test_data/videos/smartlab/stream_8_top.mp4'),
+                '-fv': TestDataArg('data/test_data/videos/smartlab/stream_8_front.mp4'),
+                '-m_ta': ModelArg('smartlab-object-detection-0001'),
+                '-m_tm': ModelArg('smartlab-object-detection-0002'),
+                '-m_fa': ModelArg('smartlab-object-detection-0003'),
+                '-m_fm': ModelArg('smartlab-object-detection-0004'),
+                '-m_en': ModelArg('i3d-rgb-tf'),
+                '-m_de': ModelArg('smartlab-sequence-modelling-0001')}),
+        ],
+    )),
+
     PythonDemo(name='sound_classification_demo', device_keys=['-d'], test_cases=combine_cases(
         TestCase(options={'-i': TestDataArg('how_are_you_doing.wav'),
                           '-m': ModelArg('aclnet')}),