<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Crowd Intelligence Driven Design Framework
## Based on Perception-Retrieval Cognitive
### Mechanism

Chen Zheng<fnref n="1" />,<fnref n="2" />(✉), Kangning Wang<fnref n="1" />, Tengfei Sun<fnref n="1" />, and Jing Bai<fnref n="1" />

chen.zheng@nwpu.edu.cn

**Abstract.** Currently, the use of crowd intelligence in which the knowledge from different disciplines is integrated for complex product design has attracted increasing attention from both academia and industry. However, the multi-modal, multi-temporal and multi-spatial characteristics of multi-disciplinary knowledge hinder its implementation. The perception-retrieval cognitive mechanism of human beings’ brain shows its unique advantages in the cognitive process of multi-modal, multi-temporal and multi-spatial knowledge, and can quickly integrate external information and retrieve memory. In order to solve the problems of low efficiency and poor acquisition accuracy of multi-disciplinary knowledge, inspired by the brain’s perception-retrieval cognitive mechanism, this paper adopts a crowd intelligence-drive to achieve efficient integration, dynamic storage and real-time acquisition of multi-disciplinary knowledge.

First, a deep survey relating to the current research studies on knowledge-based engineering approaches and the perception-retrieval cognitive mechanism is conducted. Second, the brain-inspired crowd intelligence-driven design approach for complex products and the techniques that can be used as the potential solutions to each step are presented. Finally, the authors draw the conclusion and point out the future research direction.

**Keywords:** Knowledge-based engineering · Systems engineering · Product design · Perception-retrieval cognitive mechanism · Crowd intelligence

## 1 Introduction

With the development of technologies such as the Internet of Things and big data in today’s era, product has become more and more complex, and the use of crowd intelligence in which knowledge from various disciplines is integrated for the complex product design has become the current development trend [1]. However, if the multi-modal, multi-temporal and multi-spatial characteristics of multi-disciplinary design knowledge have not been considered during the knowledge-based engineering (KBE) process for

<footnote n="1">¹ School of Mechanical Engineering, Northwestern Polytechnical University, 127 West Youyi Road, Xi’an 710072, Shaanxi, People’s Republic of China</footnote>

<footnote n="2">² CSSC Huangpu Wenchong Shipbuilding Company Limited, Guangzhou 510715, People’s Republic of China</footnote>
complex products, the design efficiency and quality will be negatively affected, which
may lead to the project delay or design failure [2].

Currently, the research field of neuroscience has brought great inspirations to the artificial intelligence, information engineering and other fields [3, 4]. According to cognitive psychology, biological cognition is a process of continuous perception of various information in the environment, and comparing and retrieving it with existing memory [5]. This perception-retrieval cognitive mechanism of human beings' brain has formed the unique natural advantage of organisms in the cognitive process of multi-modal, multi-temporal and multi-spatial information or knowledge [6, 7]. Inspired by brain's perception-retrieval cognitive mechanism, this paper proposes a human-like knowledge organization, integration and acquisition approach to support KBE process for complex products.

This paper is organized as follows. Section 2 presents current research studies on knowledge-based engineering approaches and perception-retrieval cognitive mechanism. Section 3 introduces the brain-inspired crowd intelligence-driven design approach for complex products and the techniques which can be used as the potential solutions to each step of the proposed design approach. Section 4 draws the conclusion and proposes the future research.

2 Literature Review

2.1 Knowledge Based Engineering Approaches for Complex Product Design

KBE is an automated process of identification, acquisition, and re-use based on design
knowledge, and has been widely used to promote the rapid design of products [8]. Poko-
jski et al. proposed a KBE approach which tries to integrate the knowledge from design-
ers, users, operators, etc. to achieve the multi-disciplinary integrated design for complex
products [9]. Johansson et al. developed a KBE framework to combine the knowledge
relating to information interaction, quality control and design evaluation [10]. Camarillo
et al. presented a KBE approach which uses case-based inference to push similar cases
to designers and resolve problems encountered by integrating multi-disciplinary knowl-
edge of stakeholders during the entire product life cycle [11]. However, the multi-modal
(i.e., design knowledge represented in different modalities such as natural language,
video, image, etc.), multi-temporal (i.e., design knowledge proposed in different stage
of the product lifecycle such as conceptual design, detailed design, manufacturing, main-
tenance, quality control stages, etc.) and multi-spatial (i.e., design knowledge proposed
by different stakeholders, such as designers, users, operators, etc.) characteristics of
design knowledge affect the crowd intelligence decision-making, because the informa-
tion from different sources may conflict with each other and therefore negatively affect
the reliability of crowd intelligence.

Therefore, the traditional KBE technology needs to be transformed to adapt to the
new product design requirements.

## 2.2 Perception-Retrieval Cognitive Mechanism

Cognitive psychology believes that biological cognition is a process of continuous per-
ception of various information in the environment through the senses, and retrieval of it
compared with the existing memory. Under this perception-retrieval cognitive mechanism, the brain can quickly organize and integrate information with different modalities and spatiotemporal characteristics in an optimal way, and achieve a fast and accurate memory retrieval based on external information, thus demonstrating the inherent advantages in processing the multi-modal, multi-temporal and multi-spatial complex correlations and providing inspirations for the proposed research on the efficient organization and accurate acquisition of design knowledge for complex product design. The existing studies on perception-retrieval cognitive mechanism will be presented hereafter.

**Perception Process Based on Multi-sensory Integration:** McGurk and MacDonald first proposed the concept of biological multi-sensory integration, arguing that the information from different modalities such as image, text, sound, touch, etc. can be effectively integrated in certain areas of the brain to form unified, coherent and stable perceptual information [12]. In response to this phenomenon of sensory information integration, researchers have carried out research work in two directions, i.e., psychophysics and neuroanatomy.

Psychophysics focused on the relationship between stimulus information and sensation in the process of multi-sensory integration from the macroscopic behavior of biology. Tenenbaum et al. proposed a multi-sensory integration model based on Bayesian inference [13]. The visual and auditory integration experiment conducted by Battaglia et al. [14], the visual and haptic integration experiment by Ernst et al. [15], and the visual and vestibular signal integration experiment by Hou et al. proved the effectiveness of this model in the process of multi-sensory integration [16].

Neuroanatomy starts from the microscopic nerve cell level and studies the biological multi-sensory integration mechanism. Quiroga et al. reported for the first-time multi-modal nerve cells that can respond to both image and text modal information [17]; Stein and Meredith et al. reported multi-modal nerve cells that can simultaneously process vestibular and visual signals [18]. Based on these studies, Rowland et al. proposed a multi-sensory integration model at the level of individual nerve cells [19].

In order to establish a unified multi-sensory integration model at macro and micro levels, it was found that the process of multi-sensory integration was no longer regarded as the result of the action of single multi-modal nerve cells, but was realized by the collective action of nerve cell populations in specific regions of the brain [20–22]. Beck et al. proposed a centralized framework for the multi-sensory integration model [23]. Gu et al. proved through experiments that different brain regions can simultaneously participate in the same multi-sensory integration process. In addition to dorsolateral superior temporal, ventral parietal region [24], the frontal eye field [25] and the visual posterior sylvian area [26] can also participate in the integration of visual signals and vestibular signals. Based on this discovery, Zhang et al. proposed a distributed multi-sensory integration model (Fig. 1). In this model, different multi-sensory integration brain regions estimate the stimulus information according to the input they receive, and then send their estimates to other brain regions. Finally, each brain area can integrate multiple inputs, resulting in more accurate estimation of stimulus information [27].

**Decision-making Process Based on Memorial Retrieval.** The process of memorial retrieval is completed under the combined action of short-term and long-term memory.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Distributed multi-sensory integration model.</figcaption></figure>

Short-term memory has little information and short storage time, but it is the main mod-
ule to complete the complex reasoning and calculation process in the decision-making
process. Long-term memory has a large amount of storage information and a long stor-
age time [28]. Long-term memory can quickly recall the memory content associated
with the received information [29] and can be updated and modified autonomously [30].
The Atkinson-Shiffrin memory model [31] proposed by Atkinson and Shiffrin and the
Working memory model proposed by Baddeley [32] have both explained the memorial
retrieval mechanism: after the process of multi-sensory integration, the stimulus of envi-
ronmental information is stored in the form of short-term memory in the temporal lobe
of the brain, and it is compared with the prior knowledge in long-term memory to infer
the state of the outside world, so as to actively complete the final decision.

On the basis of the above mechanism, Damasio et al. further proposed that memory
information is expressed and stored in the form of vectors after a large number of neu-
roanatomical experiments [33]. Shiffrin proposed the retrieving effective from memory
(REM). When memory retrieval occurs, the received short-term memory feature vector
matches the stored long-term memory feature vector, and Bayesian decision-making is
used to calculate whether the received information has been learned. If the information
has been learned, the decision is made using the existing experience; otherwise, it is
considered as new information and stored in long-term memory. Jiang et al. proposed a
method of learning, storing and extracting visual images based on memory recall mode
[34].

## 2.3 Summary of Literature Review

In order to take advantage of the crowd intelligence for the design of complex products,
the multi-modal, multi-temporal and multi-spatial knowledge needs to be integrated,
stored and acquired. By using the perception-retrieval cognitive mechanism based on
multi-sensory integration and REM, which enables the brain to continuously receive and
rapidly organize and integrate external multi-modal, multi-temporal and multi-spatial
information in an optimal way, and accurately achieve retrieval of memory according
to the external information[35, 36], it is possible to realize the efficient integration,
dynamic storage and real-time acquisition of design knowledge, thus achieving a crowd
intelligence-driven design for complex products.

# 3 Crowd Intelligence Driven Design Framework Based on Perception-Retrieval Cognitive Mechanism

In order to solve the problems of low efficiency and poor accuracy in the organization and acquisition of the multi-modal, multi-temporal and multi-spatial knowledge during the design process of complex products, the framework of perception-retrieval cognitive mechanism was designed from the following three aspects.

## 3.1 Organization and Integration of Multi-modal, Multi-temporal and Multi-spatial Knowledge

The knowledge adopted for the design of complex products has the characteristics of multi-modal, so the first step to achieve the crowd intelligence-driven design should be the organization and integration of the different design knowledge.

According to the multi-sensory integration process in the perception-retrieval cognitive mechanism, the features of the design knowledge can be extracted. First, the structured knowledge represented in OWL, RDF and other formats can be directly extracted. Second, the semi-structured knowledge (represented by XML or JSON) which cannot express semantic information explicitly, should be extracted semantic information by analyzing the hidden in data tags and element structures, and an OWL ontology document and description can be constructed to represent the semi-structured knowledge structure. Third, the unstructured knowledge represented by natural language, pictures or videos is identified using methods such as Polyglot [37], Mask R-CNN [38] or LSTM [39], and feature extraction is carried out in combination with the semantic relationship between the identified entities. After extracting the semantic features of the design knowledge, a semantic network with different structures can be formed, which needs to be semantically aligned. Semantic alignment is accomplished through distance-based semantic similarity, which measures the location of knowledge entities in the design ontology database. The ontology library is based on OntoSTEP in the field of mechanical design, ORA in the field of robot design or SIARAS in the field of manufacturing. Once the semantic alignment is completed, a unified representation of the design knowledge should be provided. Knowledge graph is a semantic network with graph data, which uses nodes and edges in graph structure to express knowledge entities and their relations. However, knowledge graph can only be used to express static and data-oriented knowledge, and cannot express the multi-modal, multi-temporal and multi-spatial characteristics of knowledge in KBE. Therefore, it is necessary to construct a multi-modal dynamic knowledge graph based on the knowledge graph.

At the same time, knowledge provided by different sources may contain noise, redundancy or even conflicting knowledge, which must be processed and integrated to form a semantically unified and coherent knowledge representation (Fig. 2(a)). The attention mechanism in the multi-sensory integration model proposed by Tenenbaum is used to filter and sift through noisy, redundant or conflicting information.
## 3.2 Storage and Update of Multi-modal, Multi-temporal and Multi-spatial Knowledge

According to the REM model, human long-term memory is stored in the form of feature vectors. Therefore, the knowledge graph composed of knowledge in KBE should be represented in the form of vector. Once the multi-modal dynamic knowledge graph has been constructed, and the knowledge nodes and their relations in the graph should be expressed in the form of feature vectors. Knowledge graph embedding technology can directly map the knowledge entities and their relations in the multi-modal dynamic knowledge graph into the low-dimensional vector space to realize the feature coding of knowledge. Feature vector can be obtained by using the knowledge graph embedding technology based on TransR model.

After the feature vector is obtained, the feature vector should be stored. In order to reduce the computing and searching time, according to the distributed multi-sensory integration model of brain long-term memory, the knowledge in different disciplines is featured by clustering and stored in different knowledge modules (Fig. 2(c)). Graph attention networks is used to complete the clustering of design knowledge.

Large-scale knowledge in KBE shows a high dependence on time and space, so the knowledge stored in KBE needs to be in a dynamic form of continuous renewal. According to the perception-retrieval cognitive mechanism, the newly-received knowledge is retrieved using the knowledge subgraph coding algorithm that can be developed based on the TransR model, and then compared with the existing knowledge for storage and update. Since only the subgraphs with changes are encoded, the computational load of the encoding in the process of knowledge updating can be greatly reduced.

## 3.3 Push of Multi-modal, Multi-temporal and Multi-spatial Knowledge

Referring to the short-term memory mode in the REM, the existing knowledge is called and matched to complete the push of knowledge. The research on the praxeology shows that when design participants conduct the design work, their behavior patterns are not chaotic, but have their own rules. When analyzing the behavior patterns of operators to determine whether they need knowledge, the context aware computing is adopted to determine whether they need knowledge through real-time perceptual monitoring of their own behavior and software operation. At present, the context aware computing technology based on information communication, sensors and machine learning is very mature, which provides data and technical support for signal acquisition and processing in the process of behavior pattern recognition.

After confirming that operators need knowledge, it is necessary to further acquire their knowledge needs and judge what knowledge they need. However, the traversal method will consume a lot of computing time of the system in the mass and miscellaneous crowd knowledge, which seriously affects the real-time performance of knowledge push. Nowadays, inferential methods based on ontology and rules have been widely applied in the field of information, and various inference machines supporting ontology and rules have also been developed, such as Jena, Jess and Racer inference machines, which can provide support for the reasoning process required. Therefore, a semantic inferential model is used to locate the required knowledge quickly.
In order to reduce the time spent in knowledge search, the knowledge feature vector stored in the KBE is retrieved and compared with the knowledge requirement feature vector after the system is quickly positioned using probability-based reasoning. The retrieval and comparison between the knowledge feature vector in the knowledge module and the knowledge demand feature vector can be regarded as the process of calculating the likelihood of two equal-dimensional vectors. Therefore, Bayesian formula can be used to calculate their likelihood and complete the final matching of knowledge (Fig. 2(b)).

<figure><img src="image_3.png" /><figcaption>Fig. 2. Crowd intelligence driven design of perception-retrieval cognitive mechanism.</figcaption></figure>

## 4 Conclusion

Considering the limitations of current KBE approaches, the authors propose a crowd intelligence-driven framework based on the brain's perception-retrieval cognitive mechanism for the organization, storage and push of multi-modal, multi-temporal and multi-spatial knowledge. This mechanism solves the problems of low organizational efficiency and poor acquisition accuracy of crowd design knowledge in different modalities at multiple time and space scales, and will provide theoretical basis and application prospect for promoting the intelligent design of advanced aviation manufacturing equipment and complex products such as aerospace, ships and automobiles in the future. Especially in the field of advanced aviation manufacturing equipment, the application in the field of manufacturing equipment is promoted by integrating process knowledge, product knowledge, equipment knowledge and program control knowledge commonly used in the development of manufacturing equipment.

Future research can be generally divided into two parts. First, knowledge graph has become a widely adopted knowledge representation method. The multi-modal dynamic knowledge graph proposed in this paper introduces new factors τ. On the one hand, on the basis of head node(h), relation node(r), and tail node(t), τ is integrated to realize the time correlation of the knowledge in the full-time domain. On the other hand, the modal and spatial features cannot only be integrated into the dynamic knowledge graph as the
tail node of the knowledge graph, but also form a representational spatial relationship (hasProvider) and modal relationship (hasDescription or hasImage) with the entity head node. At the same time, τ represents the update cycle of crowd design knowledge. When a new integrated knowledge graph is received, the new knowledge graph is retrospectively compared with the previous knowledge graph.

Second, the existing TransE, TransH, TransR, etc. are used as plane distance models, and their corresponding algorithms are very mature. Compared with other plane distance models, the TransR model not only solves the complex one-to-many, many-to-one and many-to-many relationships between the head and tail nodes, which cannot be realized by TransE model, but also improves the semantic expression ability of TransH model for relationships between knowledge entities. The principle of the proposed algorithm can be summarized as follows. Firstly, the low-dimensional vectors is used to initialize knowledge entities (head node (h) and tail node (t)) and their relationship (r), and the positive and negative training samples consisting of (h, r, t) can be constructed. Secondly, the TransR model based on plane distance to define the scoring function fr(h, t) is used to calculate the total loss value of the feature vector for positive samples and negative samples. Finally, taking the minimum total loss value as the optimization goal, the feature vector of knowledge in KBE is obtained through continuous calculation.

**Acknowledgement.** This project is supported by the National Natural Science Foundation of China (Grant No. 51805437) and the Fundamental Research Funds for the Central Universities (Grant No. 31020210506005).

## References

1. Jeong, Y., Pan, Y., Rathore, S., Kim, B., HyukPark, J.: A parallel team formation approach using crowd intelligence from social network. Comput. Hum. Behav. **101**, 429–434 (2019). https://doi.org/10.1016/j.chb.2018.07.018

2. Li, W., et al.: Crowd intelligence in AI 20 era. Front. Inf. Technol. Electron. Eng. **1**(18), 15–43 (2017). https://doi.org/10.1631/FITEE.1601859

3. Poo, M., Du, J., Ip, N.Y., Xiong, Z.-Q., Xu, B., Tan, T.: China brain project: basic neuroscience, brain diseases, and brain-inspired computing. NeuroView **92**(3), 591–596 (2016). https://doi.org/10.1016/j.neuron.2016.10.050

4. Pan, Y.: Heading toward artificial intelligence 2.0. Engineering **2**(4), 409–413 (2016). https://doi.org/10.1016/J.ENG.2016.04.018

5. Adolphs, R.: Cognitive neuroscience of human social behaviour. Nat. Rev. Neurosci. **4**(3), 165–178 (2003). https://doi.org/10.1038/nrn1056

6. Zhao, X., Chen, H., Xing, Z., Miao, C.: Brain-inspired search engine assistant based on knowledge graph. IEEE Trans. Neural Netw. Learn. Syst. (2021). https://doi.org/10.1109/TNNLS.2021.3113026

7. Wang, J., Cheng, R., Liao, P.-C.: Trends of multimodal neural engineering study: a bibliometric review. Arch. Comput. Methods Eng. **28**(7), 4487–4501 (2021). https://doi.org/10.1007/s11831-021-09557-y

8. Zheng, C., et al.: Knowledge-based program generation approach for robotic manufacturing systems. Rob. Comput.-Integrated Manuf. **73**, 102238 (2022). https://doi.org/10.1016/j.rcim.2021.102242
9. Pokojski, J., Szustakiewicz, K., Woźnicki, Ł., Oleksiński, K., Pruszyński, J.: Industrial application of knowledge-based engineering in commercial CAD/CAE systems. J. Ind. Inf. Integrat. 25 (2022). https://doi.org/10.1016/j.jii.2021.100255

10. Johansson, J., Contero, M., Company, P., Elgh, F.: Supporting connectivism in knowledge-based engineering with graph theory, filtering techniques and model quality assurance. Adv. Eng. Inf. 38, 252–263 (2018). https://doi.org/10.1016/j.aei.2018.07.005

11. Camarillo, A., Ríos, J., Althoff, K.D.: Knowledge-based multi-agent system for manufacturing problem solving process in production plants. J. Manuf. Syst. 47, 115–127 (2018). https://doi.org/10.1016/j.jmsy.2018.04.002

12. McGurk, H., MacDonald, J.: Hearing lips and seeing voices. Nature 264, 746–748 (1976). https://doi.org/10.1038/264746a0

13. Tenenbaum, J.B., Griffiths, T.L., Kemp, C.: Theory-based Bayesian models of inductive learning and reasoning. Trends Cogn. Sci. 7(10), 309–318 (2006). https://doi.org/10.1016/j.tics.2006.05.009

14. Battaglia, P.W., Jacobs, R.A., Aslin, R.N.: Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A 7(20), 1391–1397 (2003). https://doi.org/10.1364/JOSAA.20.001391

15. Ernst, M.O., Banks, M.S.: Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002). https://doi.org/10.1038/415429a

16. Hou, H., Zheng, Q., Zhao, Y., Pouget, A., Gu, Y.: Neural correlates of optimal multisensory decision making under time-varying reliabilities with an invariant linear probabilistic population code. Neuron 5(104), 1010–1021 (2019). https://doi.org/10.1016/j.neuron.2019.08.038

17. Quiroga, R.Q., Reddy, L., Kreiman, G., Koch, C., Fried, I.: Invariant visual representation by single neurons in the human brain. Nature 435, 1102–1107 (2005). https://doi.org/10.1038/nature03687

18. Stein, B.E., Meredith, M.A.: The Merging of the Senses. The MIT Press, Cambridge (1993)

19. Rowland, B.A., Stanford, T.R., Stein, B.E.: A model of the neural mechanisms underlying multisensory integration in the superior colliculus. Perception 10(36), 1431–1443 (2007). https://doi.org/10.1068/p5842

20. Beck, J.M., et al.: Probabilistic population codes for Bayesian decision making. Neuron 6(60), 1142–1152 (2008). https://doi.org/10.1016/j.neuron.2008.09.021

21. Gao, C., Weber, C.E., Wedell, D.H., Shinkareva, S.V.: An fMRI study of affective congruence across visual and auditory modalities. J. Cogn. Neurosci. 7(32), 1251–1262 (2020). https://doi.org/10.1162/jocn_a_01553

22. Gilissen, S.R., Farrow, K., Bonin, V., Arckens, L.: Reconsidering the border between the visual and posterior parietal cortex of mice. Cereb. Cortex 3(31), 1675–1692 (2021). https://doi.org/10.1093/cercor/bhaa318

23. Kuang, S., Deng, H., Zhang, T.: Adaptive heading performance during self-motion perception. PsyCh Journal 3(9), 295–305 (2020). https://doi.org/10.1002/pchj.330

24. Gu, Y., Cheng, Z., Yang, L., DeAngelis, G.C., Angelaki, D.E.: Multisensory convergence of visual and vestibular heading cues in the pursuit area of the frontal eye field. Cereb. Cortex 9(26), 3785–4380 (2016). https://doi.org/10.1093/cercor/bhv183

25. Chen, A., DeAngelis, G.C., Angelaki, D.E.: Convergence of vestibular and visual self-motion signals in an area of the posterior sylvian fissure. J. Neurosci. 32(31), 11617–11627 (2011). https://doi.org/10.1523/JNEUROSCI

26. Zhang, W.H., Chen, A., Rasch, M.J., Wu, S.: Decentralized multisensory information integration in neural systems. J. Neurosci. 2(36), 532–547 (2016). https://doi.org/10.1523/JNEUROSCI

27. McBride, D.M., Cutting, J.C.: Cognitive Psychology Theory, process, and Methodology. SAGE Publications, Thousand Oaks (2018)
28. Eysenck, M.W., Keane, M.T.: Cognitive Psychology. Psychology Press, London (2015). https://doi.org/10.4324/9781315778006

29. Lee, J.L., Nader, K., Schiller, D.: An update on memory reconsolidation updating. Trends Cogn. Sci. 7(21), 531–545 (2017). https://doi.org/10.1016/j.tics.2017.04.006

30. Eysenck, M.W., Brysbaert, M.: Fundamentals of Cognition. Routledge, New York (2018)

31. Baddeley, A.: Working memory and conscious awareness. In: Theories of Memory, pp. 11–28(2019)

32. Damasio, A.R.: The brain binds entities and events by multiregional activation from convergence zones. Neural Comput. 1(1), 123–132 (1989). https://doi.org/10.1162/neco.1989.1.1.123

33. Jiang, Y., Wang, Y.: Application of REM memory model in image recognition and classification. CAAI Trans. Intell. Syst. 3, 310–317 (2017)

34. Seilheimer, R.L., Rosenberg, A., Angelaki, D.E.: Models and processes of multisensory cue combination. Curr. Opin. Neurobiol. 25, 38–46 (2014). https://doi.org/10.1016/j.conb.2013.11.008

35. Bertin, R.J.V., Berthoz, A.: Visuo-vestibular interaction in the reconstruction of travelled trajectories. Exp. Brain Res. 1(154), 11–21 (2004). https://doi.org/10.1007/s00221-003-1524-3

36. Gu, Y., Angelaki, D.E., DeAngelis, G.C.: Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci. 10(11), 1201–1210 (2008). https://doi.org/10.1038/nn.2191

37. Nystrom, N., Clarkson, M.R., Myers, A.C.: Polyglot: an extensible compiler framework for java. In: Hedin, G. (ed.) CC 2003. LNCS, vol. 2622, pp. 138–152. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36579-6_11

38. He K, Gkioxari G, Dollár P, et al.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

39. Greff, K., Srivastava, R.K., Koutník, J., et al.: LSTM: a search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 10(28), 2222–2232 (2016). https://doi.org/10.1109/TNNLS.2016.2582924