2024/3/31

Background12

 Developing 6G technologies[1]Multimodal Object Recognition
Using Bayesian Attractor Model
For 2D and 3D Data Connecting various types of sensors[2,3,4]
 RGB Camera, LiDAR sensor, Infrared sensor
 Analysis at edge and cloud Exploring 6G Use Case Robot control with digital twin Humans and robots cooperate in logistics warehouse Stochastic robot control for safety and efficiency

H. Ando, D. Kominami, R. Seki, H. Shimonishi and M. Murata

 Stochastic recognition techniques are essential

Osaka Univ.

Example of co-operative working

27th Conference on Innovation in Clouds, Internet and Networks
Workshop on 6G Network Use Cases and Verticals 6GN[1] H. Viswanathan and P. E. Mogensen, “Communications in the 6g era,” IEEE Access, vol. 8, pp. 57 063–57 074, 2020.
[2] X. Ding, J. Guo, Z. Ren, and P. Deng, “State-of-the-art in perception technologies for collaborative robots,” IEEE Sensors Journal, vol. 22, no. 18, pp. 17 635–17 645, 2022.
[3] Wu, Hai, et al. "Virtual Sparse Convolution for Multimodal 3D Object Detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[4]Jiao, Yang, et al. "MSMDfusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Mar.11th-14th, 2024, Paris.

1

2

Our Previous Work: Overview

Feature Extraction by Siamese-RPN

3

Feature Extraction

Object Recognition

 Object recognition method inspired by brain information processing mechanism

4Siamese RPN

 One-shot object detection method[6][5] Calculates Posterior probability density (Confidence) about the observed object

 Video features

 Realizes Uncertainty-aware object recognition using confidence levels and enable risk assessment

 Location features : Extract features by combining position of Bounding Box and depth images: Extract features by CNN-based extraction as the region (Bounding Box) where an object exists

 Center of Bounding Box (x, y) and depth (z) Remaining Issue: Dependencies between modalities

 Dependencies arise in the assumptions of Siamese RPN feature extraction before integration

 Dependencies arise between recognition results before integration due to shared feature extraction parts[5]R. Seki, D. Kominami, H. Shimonishi, M. Murata, and M. Fujiwaka, “Multi-object recognition method inspired by multimodal information processing in the human brain,” in 2022 IEEE Globecom Workshops.

[6]B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High Performance Visual Tracking With Siamese Region ProposalNetwork,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 201834

Unimodal Object Recognition by Bayesian Attractor Model

Multimodal Object Recognition by Bayesian Causal Inference5

BAM[7]BCI[8] Multimodal object recognition using BCI Model that integrates different modalities

 Unimodal Object Recognition using Brain-inspired model considering uncertainty

 Reduces recognition error by leveraging different modalities

 Represents human decision state using state-space model that has fixed points called attractors

 Assesses if the observation is focused on the same object and establish the weights accordingly Attractors store the characteristics of each object

 Calculates confidence by integrating the probability distributions obtained from each modality

 Updates its state according to the observation and recognize objects stochastically Support decision-making by considering uncertainties associated with modalities

 Posterior probability density (confidence) on each attractor is derived as the certainty of recognition

 Construction of risk-aware systems is made possible by recognition with confidence[8] K.P. K¨ording, U. Beierholm, W.J. Ma, S. Quartz, J.B. Tenenbaum, and L. Shams, “Causal Inference in Multisensory Perception,” PLOS ONE, 2007.

[7] S. Bitzer, J. Bruineberg, and S.J. Kiebel, “A Bayesian Attractor Model for Perceptual Decision Making,” PLOS Computational Biology, 2015566

2024/3/31

Contribution in Present Work

Difference Between Previous and Present Work7

 Problem:Dependencies before multimodal integration

8

 Previous Work

 Use the information extracted from RGB images by Siamese RPN when processing depth images

 Video Modality Dependencies arise between recognition results before the integration of multimodal data

 RGB images
 Siamese RPN[6] The reduction in recognition in the video modality results in decreased recognition in the location modality Location Modality

 Deterioration of dependability

Previous Work Depth images

 Contribution:Resolve dependencies in the feature extraction

 Combining with Bounding Box

 Employ point clouds as a modality capable of standalone analysis, independent from video modality

 Present Work

 Adopt PointNet[9] for semantic segmentation

 Video Modality

 Construct to merge video and location modalities following their separate recognition

 RGB images

 Improvement of dependability

 Siamese RPN[6] Location Modality

Present Work

 3D Point Clouds PointNet[9]Comparing Two Methods[9] C.R. Qi, H. Su, K. Mo, and L.J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” Proceedings of the IEEE conference on computer vision and pattern recognition, 201778

Present Work: Unimodal Object Recognition by PointNet

9Evaluation Settings

10 Create dataset using an RGB-D camera and a LiDAR sensor
 Time-series data consisting of RGB-D images and 3D point clouds with labels
 A scene where a human and a robot are performing cooperative work indoors

 Tasks for object recognition from multimodal sources Targets : Worker / Robot Task

: Estimate what given feature corresponding to each object represents

PointNet with BAM
Our present work using Bayesian Attractor Model for RGB and Point Clouds

WorkerRobot

WorkerRobot Recognition model combining BAM [7] and PointNet [9] Utilize 3D centroids of objects instead of location based on depth images and Bounding BoxIntel® RealSense
Depth Camera D455[10]1. Performs semantic segmentation of point clouds using PointNet
2. Acquires labelled point clouds and calculates centroids of objectsTask[10] https://www.intelrealsense.com/depth-camera-d455/[11]https://www.livoxtech.com/jp/avia

9RobotLivox Avia[11] Recognition with 3D centroids using BAM
 Memorize the centroids at 1st frame in BAM, compare observation with them, and recognize an object with confidence

Worker10

Evaluation Metrics

11

Quantitative Evaluation: Misprediction of Bounding Box

 Inputs and outputs

Worker Input (video) : image features representing each object per frame Input (location) : position features representing each object per frame

Confidence

HighWorker Output : confidence for each objectLow

Robot

Worker Blue: Worker confidence, Orange: Robot confidence

Robot Quantitative and qualitative evaluation
 Quantitative evaluation: Transition of confidence How recognize the objects in each modality and integrated one

 Qualitative evaluation: Precision

Frame
Example of output
when observing Worker

Worker

Robot
Video ( RGB )Robot
Integrated

Confidence get down
Location ( Depth )Previous Method for RGB and Depth Images

Robot
Integrated

WorkerRobot

Confidence keep highLocation ( PC )Present Method for RGB Images and Point Clouds

Transition of Confidence (Target: Worker)

 Integration when one modality's recognition is high and the other's is low.
 A decrease in recognition in the video modality leads to a decline in recognition in the location modality as well.
previous method does not improve the recognition by integration

 Percentage of objects that are truly correct when an object is recognized Calculated by assuming that the object with the highest confidence as recognized

11Worker

Video ( RGB ) Example of output where worker features are given as input
 Horizontal axis: Time [frames], Vertical axis: Confidence

WorkerRobot

12

 When the confidence of the worker is low in the video modality and high in the location modality,
present method keeps the confidence of the worker in integrated result high, even if robots are also recognized122024/3/31

Qualitative Evaluation: Multimodal Object Recognition1314

 Proposed method

Result on Precision
ModalityConclusion

WorkerRobot

Single modality (video)

1.000

0.594

Single modality (depth)

1.000

0.806

Single modality (location)

0.788

1.000

Video-based Multi-modalities (with depth)

1.000

0.784

Video-based Multi-modalities (with location)

0.936

0.869

 Multimodal object recognition method for 6G use case
 Develop unimodal object recognition using point clouds with confidence Improve our previous multimodal object recognition method using brain mechanisms

 Experimental results Capture and test the environment for our use case using an RGB-D camera and a LiDAR sensor
 Observe improvement of recognition by integrating recognition results

 Future work Expand recognition targets

 The precision for both targets are improved by multimodal integration

 2-class classification and one object in each class, assuming both exist in the frame in current experiments

 Single modality reduces the certainty of one side

 For practical use, it is necessary to support multi-class multi-objects.

 Multimodal integration results in high precision score for both targets

 Propose a method to associate results from different modalities Association of recognition results is given In current work

13

14

Appendix15

 Bayesian Attractor Model[7]Bayesian Attractor Model (BAM)[7]16 A model combining Attractor model with Bayesian theory
 A model inspired by the process through which the brain makes decisions based on sensory organ information

 Bayesian Causal Inference[8] Fusion of the attractor model for memory and the Bayesian estimation model for state updating Produce recognition outcomes along with confidence based on continuous observations of what is being observed PointNet[9] Attractor model Example of Bounding Box Detection Employing Siamese RPN A model of recognition grounded in the brain's mechanisms of memory and cognition
 Place fixed points (attractors) on the state space that correspond one-to-one with the objects of observation

 Evaluation Results: Unimodal Object Recognition

 Internal state changes in response to input

 If inputs for the same target are sustained, it converges to the attractor that represents that target

 Evaluation Results: Multimodal Object Recognition

 Bayesian theory

 Quantitative Evaluation

 Calculate the posterior probability using prior probabilities and observations
 Compile information continually upon receiving new inputs to revise decisions

15

16

Bayesian Causal Inference (BCI)[8]17

PointNet[9]18 A statistical model combining Bayesian theory and causal inference A model inspired by the process through which the brain makes decisions based on multimodal information
 Infer whether objects recognized by two different modalities are the same and integrate Combine modalities with different advantages to provide more accurate predictions
 Evaluate the likelihood that inputs from various modalities stem from a single external occurrence Based on evaluated causal links, determine if sensory inputs should be merged or regarded independently

 Formula for generating confidence

PointNet Architecture[9] 𝑐 , 𝑐 :Confidence in video and location modalities

 𝑃 𝑐 ,𝑐 ∣ 𝐶 = 1 = 𝑃 𝑐 ,𝑐 ∣ 𝐶 = 2 =0.5 + 𝜎𝜎
10

if 𝐿 = 𝐿 ,otherwise,if 𝐿 ≠ 𝐿otherwise Deep learning framework for processing 3D point clouds
 𝜎= 0.5𝜎 max 𝑐 𝜎= 1/ 1 + exp −𝑥 − 𝜆 Apply the same Multilayer Perceptron (MLP) to each point, followed by the application of Max-Pooling+ 0.5𝜎 max 𝑐

 Produces the same output, independent of the order in which the point cloud is inputted Utilize a subnet called T-Net

 𝐿 = argmax 𝑐

 Produces an affine transformation matrix for the input, allowing for adjustments such as rotation and shifting 𝐿 = argmax 𝑐 Utilizing global features
 Integrate local and global features and apply MLP for the realization of semantic segmentation

1718

2024/3/31

Example of Bounding Box Detection Employing Siamese RPN

19

Evaluation Results: Unimodal Object Recognition20

 Unimodal Object Recognition
 Use the term "depth modality" to distinguish the location modality using point clouds that employs depth images
 Observe the correlation between video and depth modalities, which is a challenge in previous work
Video Modality

Depth Modality

Location Modlaity

Worker（=Object 1）

WorkerRobot

Robot（=Object 2）

Evaluation results of unimodal object recognition

1920

Evaluation Results: Multimodal Object Recognition

21

Quantitative Evaluation

 Multimodal Object Recognition

Confidence keep highWorker When using depth images, the parts where correlation was observed have not been corrected

WorkerRobot When using point clouds, the problem has been resolvedVideo Modality+Depth Modality

22

Video Modality+Location Modality

Depth Modality+Video Modality

Location Modality+Video Modality

Confidence keep highWorkerVideo ( RGB )WorkerWorker（=Object 1）

Worker

Robot
Video ( RGB )Robot
Integrated

Robot

WorkerRobot
Integrated

Robot
Location ( Depth )Previous Method for RGB and Depth Images

Location ( PC )Present Method for RGB Images and Point Clouds

Transition of Confidence (Target: Worker)

 Integration when both modalities produce recognition results with the same trend When the confidence of the worker in the video and location modalities are high,the confidence of the worker in integrated result is also highRobot（=Object 2）

 When the confidence of the robot in the video and location modalities are low,
the confidence of the worker in integrated result is also low

Evaluation results of multimodal object recognition

2122

Quantitative Evaluation: Misrecognition in Previous Method
WorkerWorker

Robot

Worker

Robot

Worker

Robot
Video ( RGB )Worker23

Video ( RGB )Robot
Integrated

Confidence get down
Location ( Depth )Previous Method for RGB and Depth Images

WorkerRobot

Robot
Integrated

Confidence keep highLocation ( PC )Present Method for RGB Images and Point Clouds

Transition of Confidence (Target: Worker)

 Integration when both modalities confidently provide different recognition results
 Bounding Box is right in the video modality, but the perception continues to be wrong
 Misrecognition occur in the depth modality where they pass each other in the latter part of the scene
 It is clearly recognizable by using centroids in location modality

23