Sensory Resilience based on Synesthesia
Authors: Eric Platon
Tom Sonoda
Venue:  (2021)
Keywords: perception
resilience
robotics
synesthesia

Abstract: Situated cognition depends on accessing environmental state through sensors. Engineering and cost constraints usually lead to limited “pathways” where, for example, a vision sub-system only includes a camera and the software to deal with it. This traditional and rational design style entails any hardware defect on the pathway causes the system to grind to a halt until repair. We propose a “sensoriplexer” as drop-in neural component architecture to address this issue, under the common scenario of multiple sensors availability. This component architecture learns to mix and relate pathways, such that an agent facing failure in a sensory sub-system can degrade gracefully and coherently by relying on its other sub- systems. The architecture is inspired by the concept of synesthesia, and relies on statistical coupling between sensor signals. We show the benefit and limitation of the architecture on a simple shape recognition and a more complex emotion recognition scenarios.

Under review as a conference paper at ICLR 2021
SENSORY RESILIENCE BASED ON SYNESTHESIA
Anonymous authors
Paper under double-blind review
ABSTRACT
Situated cognition depends accessing environmental state through sensors. Engineering and cost constraints usually lead to limited "pathways" where, for example, a vision sub-system only includes camera and the software to deal with it. This traditional and rational design style entails any hardware defect on the pathway causes the system to grind halt until repair. We propose "sensoriplexer" neural component architecture to address this issue, under the common scenario of multiple sensors availability. This component architecture learns mix and relate pathways, such that an agent facing failure in a sensory sub-system can degrade gracefully and coherently by relying on its other sub-systems. The architecture is inspired by the concept of synesthesia, and relies on statistical coupling between sensor signals. We show the benefit and limitation of the architecture on a simple shape recognition and more complex emotion recognition scenarios.
INTRODUCTION
Situated agents embody various degrees of "SPA loops", where they Prons-Proccss-Actt repeatedly in their environment. The simplest agent senses only one dimension the environment. Sensing failure prevents it from processing and with multiple sensors can sometimes continue operating despite the loss of some sensors, but they cannot usually perform anymore the functions related to the lost sensors.
causing it to halt. More complex agents
Sensor redundancy is common solution to ensure continuing operations. Flagship projects like space rovers introduce redundant sensors, such as doubling all "hazcam" on NASA's Curiosity or CNSA's Yutu. Yet engineering and cost constraints block this option many systems. The loss of most non-critical sensors often means the agent enters "degraded" mode of operations.
Biological agents can compensate to some extent the loss of sensor, by using the other ones they are endowed with. A blind person can rely on touch to "read" visual cues encoded to Braille, or "hear" speech signed visually. Impressive technology allows learning to feel sounds on the skin [Bach-y Rita (1972). and by extension for deaf people to "listen" to conversation by translating sound into tactile patterns on their skin Novich & Eagleman (2015). The compensation capability appears to rely on brain mechanisms that relate different sensory inputs of the same object. These brain mechanisms also appear common to all, with different degree of expression Cytowic (2018). For example, we perceive an apple via (mainly) vision, touch and smell. We can relate the vision of apple to its likely smell or touch. Although the brain mechanisms are not entirely identified at the present time, the effect has been named synesthesia, sometimes ideaesthesia Jürgens & Nikolič (2012); Nikolić (2014).
This article proposes the sensoriplexer component architecture to model synesthesia-insppree mech- anisms in artificial agents. The sensoriplexer (SP) allows an agent learning relations between its "senses" for exploitation its downstream activities. After presenting work related to resilience and their limits, we present a formal model of SP and a corresponding implementation"" We use the implementation to conduct series of experiments to demonstrate and evaluate the capabilities of systems including SP. The article ends with a discussion of the results, and future work.
Informally, redundancy means here setting multiple sensors on the same environmental object. Two close-by front cameras on robot are redundant, but a front and rear camera pair is not (they are complementary). Available the supplementary materials, and on GitLah (private for now)

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
The problem at hand is of resiliency: A situated agent is resilient if it can continue operating despite a usually pre-defined range of failure modes. The pre-definition covers explicit problems the agent must deal with (e.g. critical safety, guaranteeing a car stops), classes of problems (e.g. liveness properties, absence of deadlocks), and sometimes specifications of behaviours under partial failures (e.g. a car must be steerable and stoppable even if the air conditioner fails). Resiliency is broad topic, and we frame its meaning for the extent of this article. An important property of resilience framed this way is its extent: We expect resilient agent to continue operating (as autonomously as possible), yet within compliance, such as safety guarantees.
The sensoriplexer addresses resiliency issues in situated agents, using concepts rooted in Neuroscience and Psychology. Other disciplines have inspired alternative approaches, like redundancy: The loss of a sensor triggers "switch" to a redundant one. Redundancy is a powerful mean to achieve resilience, but it is not always feasible. Aside cost issues incurred by extra sensors, some systems cannot physically accommodate any extra. Typical quadcopter drones with autopilot capabilities (seen here as situated agents) embark a camera, but they often cannot host second one, whose payload would exceed the lift capabilities. A similar example is on sensors in satellites and space rovers, where weight and available mount space are very constrained: Redundancy is reserved to safety-critical sensors only. In this paper, we do not see SP as solution to compensate advanced scientific sensors satellites, but we expect the concept of synesthesia to inspire future sensor designs for such extreme missions.
Programmable sensors offer alternative grounded in Electronics and Chemistry. Programmable packages like the CMUCan (where the sensor is not directly programmed) and sensor chips em- bedding artificial neural networks like the IMX500 from Sony provide agent designers with the capability to programmatically address resiliency issues CMU (020); Sony (2020). The trend of "intelligent sensors" is relevant the problem of resilience, yet their application remains limited under the present understanding of their capabilities. A vision sensor can adapt to lighting issues by switching frequency, but cannot relate or leverage other signals.
On a more theoretical note, we relate this work to the thread of research stemmed from the ef- ficient coding hypothesis, proposed by Barlow, and most notably the statistical representation of signals Barlow (1961). The hypothesis refers to sensory wetthways-concpeee of the environment, through areas of the brain, to their final destinations in the brain-for modeling the processing of sensory "messages". Later work shows a statistical dependency between some pathways [Simoncelli (2003). SP an adaptation and implementation of these dependencies in artificial agents. These dependencies are concretely modeled as joint distributions over the differ- ent pathways, without explicit relation to any architectural candidate area in the brain (we aim at functional equivalence only, at best).
routes from a sense
Another related theory is practopoiesis from Nikolić Nikolić (2014). This theory of "creation of actions" models general principles of abstractions, and introduces the notion of "traverse" along sensory pathways. Traverses separate representation realms, from what we qualify as low-level raw signals to higher-level representations of these signals. The higher-level representations introduce new semantics, typically meaningless at lower levels. This theory can loosely be compared with multi- layer convolutional networks in (for example) character recognition, where the first layers compile low-level glyphs with broad semantics, and last layers capture higher-level concepts of characters. Practopoiesis indicates the number of traverses necessary achieve level of representation depends on the task at hand. SP in this article models a tentative traverse from individual sensory paths to an abstract representation of objects represented by the incoming signals. We do not claim SP is a primitive traverse. Its purpose is to model and implement joint representations of objects, and, again, may not be grounded in any biological apparatus.
A final note pertains to the difference with sensor fusion, often critical to the design of systems like self-driving cars Wikipedia (2020). Sensor fusion is closely related to the functions implemented by SP, but they serve different purposes. Sensor fusion aims at decreasing uncertainty about an object, typically its position or some other property. Sensor fusion relies on adding sensors and "fusing" their data into coherent pictures. The addition continues until acceptable uncertainty is achieved for the given target. SP aims at compensating sensor failures by exploiting other sensors, í.e. decreasing uncertainty whenever removing sensors.
2

---

Under review as a conference paper at ICLR 2021
3 A SENSORIPLEXER MODEL
The environment embedding agents is, from an agent perspective, a set of N signals it can influence in M ways: Env (,, agent typically access < N environmental signals X; to accomplish its tasks. Each Y; E 1j is an "influencable aspect" of the environment. An agent influences m M aspect of the environment. We note E (E,)
( (T))j<M), where each X; E E; is sequence of signal samples. An
situated agent Ag Env consists of sets of processes P. environmental sensors and environ- mental actuators A: Ag (P, S, A), where S = (st) represents the agent sensing environmental signals E, for each corresponding 8i, and similar relation on influences. For sake of simplifying the notations, we will drop the actuators from the model, as we focus entirely on the sensory part of the agent. Also, we assume in this presentation that all environmental signals are sequences of length 1. again simplify the notations. The model focuses however exclusively on sequences (an image would simply be a sequence of length one).
Each process, sensor, and actuator is modeled a mathematical function f over multiple dimensions Vf E 3(,, m) E I D1 C C", C C" D - D2
Environmental signals perceived by the agent modeled as random variable vectors X,, with samples in E; for each signal < N. Actions in the environments are the output of the actuators, also modeled random variables Yj. with samples in for each influence < M. Samples in each_ X and Y are vectors whose dimensions depend on the signal. Here too X and Y are reduced sequences of length for the presentation of the model. The extensive form of X (Y similar) becomes: X = (m)d for sample from signal E..
In the remainder of the article, each function f is represented by matrix M and shift s, such that for a signal X, f(X) MX + s = Y. An agent functioning represented with this notation becomes set of proper compositions over S P x A. For single process, sensor and action in each corresponding set, we get Ag(X) A P o S(X), describing an agent SPA step.
3.1 SENSORIPLEXER
The sensoriplexer operates on input signals, right after sensing by the agent. Noted Sx (Sx:)ieNs we define for given agent Ag (P, A): Sx S + S, with the given properties:
Transparency: Sx;= Idg when signal is non-null. VX E E,, Sx(X)
Stability: Sx(0s) Og: rest in the absence of sensing.
Compensation: Whenever an input signal is null in S,(X) Sw. compensates using
other signals through some delegate function θ II. S + II. S₁ Vi S n, (O,.) = (((E)))44)
The θ delegate function embodies different compensation methods, which could introduce heuristics specific to a given problem. We propose a generic method in the next section.
It is important to note Sx applies to signals sensed null by the agent. This differs from the absence of signals from the environment: Sx activates on any such that (x) = 0 (null sense), even X + (non-null environment signal).
Also, the transparency and compensation requirements are opposing forces, leading to the absence of exact solution in this model. Compensation introduces signal couplings added to each "pure" signal, causing losses in transparency. In terms of linear algebra, this can be represented with:
Sx(X) = X O(X)
Where pure transparency would require the compensation term to be null. An agent endowed with sensoriplexer operates then the overall (simplified) function: Ag(X)= A o P Sr S(X).
Here too, functions here map sequences. The formulas use vectors to simplify the exposé. The model and implementation have been designed over tensors process sequences.
3

---

Under review as a conference paper at ICLR 2021
Signal
Vileo Signal
Audio Viáco
Source
Senscribieer
Audio Signal
Aud a Signal
Figure 1: (Left) Sensoriplexer implementation as a stack of auto-encoders with traverse. (Right)
Common pattern for all experimental settings.
3.2 THE θ DELEGATE FUNCTION
The delegate function operates the input signals of the agents, under the properties of the sen- soriplexer. We present here an approximate method that complies with all the properties, trying to minimize convolution with the input signals.
We define 5, for all < n as S; complemented with zeros up to c = max(dim I n). Doing so all S; have the same dimension c. We also define the agent "internal representation" IR Uign S as the ne-vector (in general, tensor) obtained by stacking objects from Uign Vi < n, S, IR (50, 3,). If represents object of the environment by vector gathering vectors from all sensor signals.
We then model the delegate function with ne X ne square matrix, where the ne size serves to apply θ to IR, and diagonal blocks of size are all zeros, so e represents the contribution to signal i from all signals except Together with formula equation [ SP performs in this model:
B12 0
B
B21
B23
Bzn
Sz(IR) = IR
IR
(2)
Bai
Ban
Where IR = (3.....,
3,)) ) is some internal representation vector, and e x block matrix By is compensation from signal j over This formulation models SP properties transparency and stability, and compensates missing senses by adding learned couplings from other signals. The last property of reversibility expected from an inverse matrix, which results from the auto-encoding overall architecture.
In terms of statisties over the different signals distributions, the SP block matrix formulation is akin to covariance matrix over random variables, minus the ne identity matrix. This representation thus captures joint distributions over the n signals integrated by the agent.
We conclude the model presentation with quick informal example θ. Let us assume S (image, sound, smell) senses for the agent, each represented by a vector. The Lapppe for an apple is vector Rapple = image + sound + smell. At some point in time the agent loses the image sense. IR becomes + sound + smell. is designed to transform IR into I Rapple using the available senses of sound and smell reproduce sense of image by minimizing distance metric (e.g. Euclidean distance in our experiments).
3.3 OPERATIONAL APPROXIMATION
We present approach based Machine Learning to approximate the θ delegate function. The approach consists of a training algorithm and a loss function. The model presented this article requires approximating the block matrices parts of the è delegate function. We perform the approxi- mation by implementing the sensoriplexer as stack of auto-encoders Hinton Zemel (1993), with the O function internal representation.
4

---

Under review as a conference paper at ICLR 2021
Figure I describes the overall architecture of the SP component. The left coloured boxes vectors each representing the input layer for a signal, e.g. yellow for image, green for sound, and red for temperature. The output of the component indicated by the colouring. The internal layers consist of an encoder collecting the concatenated inputs to feed the encoding layers. The code is then fed into the traverse component, which we model as a square matrix here to implement the block matrix indicated in equation 2 of the @ delegate. This traverse matrix is initialized as a random matrix. Last, the traverse output is fed into decoder, symmetric with the encoder, to reconstruct multi-sensory signals. The agent functionalities would then get their inputs from the decoder.
the right corresponds to the imput one one, as
The auto- uuto-encoder base implementation of SP complies with the transparency and stability properties. an approximation of the SP model, though, as an auto-encoder approximates these properties. The encoder/traverse pair is also model inspired by the work of Nikolić Nikolić (2014). It encodes raw signal inputs into "higher-level", yet more compact, internal representation reserved to the agent use. In this paper, the agent uses only the decoded representation from the traverse (the traverse code could provide direct clues to the agent-reserved for future work).
EXPERIMENTS
The purpose of the experiments is to validate both the expected resiliency from adding a sensoriplexer, and the formal model properties, which we expect learned by SP. This paper reports two comple- mentary experiments: (1) an illustrative extension to a simple shape classifier, (2) an application to emotion recognition.
The shape experiment in (1) aims at demonstrating and quantifying the benefit and cost of introducing SP upstream to a straightforward classifier. The emotion recognition experiment in (2) shows the potential and limitations of SP on a more realistic data set. It also illustrates possible uses as independent module for engineering systems, and stresses the conditions on the simple SP implementation in the frame of this work (as an auto-encoder with matrix code).
All settings adopt the same organization: We combine an audio-video SP (i.e. accepts as input image and sound frames), with an image-only classifier downstream. We then evaluate for each setting four execution scenarios. Figure [ pictures the system architecture common to all scenarios. The data source on the left produces streams of images and sound frames, fed into SP. The classifier is plugged to the image output from the sensoriplexer, and produces a class vector as output. Table [ summarizes the four execution scenarios evaluated for each setting. The loss function is then the sum of the losses for each scenario.
Table 1: Common scenarios evaluated for each setting.
Scenario Name
Description
Direct Signal (DS)
The video signal is directly input to the classifier, effectively bypassing the sensoriplexer. This scenario serves as baseline.
Only Video (0V)
Only the video signal is sent to the sensoriplexer.
Audio/Video (AV)
Both audio and video signals are sent to the sensoriplexer.
Audio Only (A0)
Only the audio signal is sent to the sensoriplexer.
The Direct Signal scenario serves as baseline, which is the "standard" use of the downstream classifier. The Only Video scenario is same as baseline, going through SP. It allows to evaluate the distortion introduced on the classifier native signal. The Audio/Video scenario adds audio through the SP to evaluate cross-signal distortion. Last, the Audio Only scenario submits audio only to the SP. It shows how much adding the SP allows the classifier becoming resilient to the loss of image. The same classifier without SP would not be applicable to the AO scenario.
Implicitly, audio and video signals are synchronized and coherent, coming from the same source. We exclude from this evaluation investigating the effect of incoherent or desynchronized output, for this paper to focus first on the potential of the approach.
5

---

Under review as a conference paper at ICLR 2021
4.1
VISUAL SHAPE CLASSIFICATION FROM SOUND
4.1.1 SETTINGS
We provide self-contained shape classifier with synthetic data set, available with the implemen- tation of the sensoriplexer. It consists simple classifier trained at identifying 4 shapes RGB frames. The data consists of generated I-second videos, showing the same shape on all frames over a uniform black background, with the spoken word for that shape as audio. Images and voices synthesized by the implementation. Shapes vary in position, size, colour. Voices vary in speed and pitch, as some examples show on fig. 3 to
and (video frames and audio spectrograms),
The code accompanying this paper allows generating datasets of any size. the reported results, have used a data set of 1000 audio/video examples, with 85% for training, 5% for validation, and 10% test holdout. We train the classifier to saturation, baseline for comparison.
4.1.2 RESULTS
Table 4.1.2
compiles the classification accuracy results the test holdout under the four scenarios. Results are based on more than runs with different seeds, with SP configured with n signals, and 32 vector length for the /R internal representation (the SP core then 64 64 matrix). Each input in the test holdout submitted four times to the system, once for each scenario. The reported results are therefore consistent and comparable.
Scenario DS
Accuracy (10-²)
Impact of SP
100%
OV
98.23%
-1.77% (against DS)
AV
97.19%
-2.81% (against DS)
Figure 2: Examples duration and pitch variability in pronouncing "square".
AO
32.60%
+7.60% (against random)
Table Shape classification accuracy results.
OV shows the introducing of SP causes an expected performance drop. This simple classifier suffers a drop of about 2%, just by introducing SP. AV shows the addition of the audio signal induces some "noise" inside SP, affecting its outputs. In particular, the video output submitted to the classifier differs more from the original input, and causes a total drop in accuracy of close 3%. AO leads a low performance of 32%, still higher than chance- 25% in this 4-class experiment.
Samples fig. G show classification results, together with the corresponding output from SP, the actual image input to the classifier. We can observe how AO achieves reconstructing images from audio, evoking the expected shape. In particular, (a) and (b) show that squares of very different sizes reconstructed alike under A0, which indicate SP has learned associate the spoken word "square" to a single, summary representation the concept. (c) shows a similar example on an ellipse, where the reconstruction from audio in AO is vague and appears like a rectangle. Last (d) shows the progressive degradation of the input across SP, where tiny circle is reconstructed with more difficulty when audio is added, illustrating the impact from multi-signal SP.
DS GT=square R=square
OV GT=square R=square
WV GT=square R=square
GT=rectangle GT=rectangle GT=rectangle GT=rectangle Rerectangle
GT=square Rerectangle
R=rectangle
R=rectangle
R=rectangle
Figure All Successes.
Figure 4: Vagueness in Audio Reconstruction.
OV GT =elllippse R-elipse
AV GT=elliss R-elipse
AO
DS GT-circle Recircle
0V GT -circle Recircle
GT-elipse Reellipse
GT-ellipse Rerestangle
GT-circle R-al ipse
GT-circle Recetanngle
Figure 5: Weak Reconstruction from Audio.
Figure 6: Successive degradation.

---

Under review as a conference paper at ICLR 2021
The AO performance shows potential with respect to resiliency. In the absence of SP, the classifier returns whatever output for a null image (or errs due to input shape mismatch). The SP allows the classifier to make some sense out of AO input signal. A robot endowed with this SP and classifier could continue operating, to some extent, despite the absence of image.
Last, the diagonal blocks of SP's traverse reported the end of training with mean of -0.00, with values ranging from -0.89 to 0.76 over all these blocks. The small mean validates a posteriori the traverse model with blocks of zeros the diagonal (the traverse is initialized with random values).
4.2
EMOTION RECOGNITION
4.2.1 SETTINGS
This experiment investigates the impact of introducing sensoriplexer on an arbitrary classifier found on the Internet, as well custom classifier created for this paper. We have selected two emotion classifiers found on GitHub-
Expi and Exp2 below-with announced respective recognition accuracy of 78% and 74% on FER 2013 data set (7 emotions) Rajaa (2019); Wu (2018). These two generic classifiers could not generalize well from their training data. So we also provide results from Exp3, a custom emotion classifier trained on fraction of our data set (8 emotions).
selected data set, which differs significantly
In this experiment, we train and evaluate the systems on the RAVDESS data set, a set facial and vocalized expressions acted by professionals Livingstone & Russo (2018). The audio-video files in this data set share a common pattern: A single actor face, centered in the view with white uniform background, speaking in silent environment. The files vary in length, how actors express emotions, and frame by frame (the video frames in the shape data set are constant). The data set is then more complex structure and richness, than the synthetic shape data set in the previous experiment.
Emotion recognition from speech requires at least 200ms of audio Pfister & Robinson (2010). The experiment creates training data by clipping the original files into 300ms-long clips-longer than the minimal requirement and short enough to meet our computational capabilities. Each example in the training procedure is then a clip and label (so all clips from the same file share the same label).
The custom classifier Exp3 has been trained on a fraction of RAVDESS itself, using a single video frame every second (videos are 30 frames per second). The purpose of this biased setting to show the impact of SP on more realistic data set, not to evaluate the classifier itself. In fact another set of experiments could evaluate the performance of jointly training an SP and downstream systems. These configurations are all valid, and we chose focusing on SP isolated component in this paper.
4.2.2 RESULTS
our results on the test holdout under the four scenarios. Results are based on more than runs with different seeds, with SP configured with 2 signals, and 64 vector length for I R. We do not report here results from runs with e ranging from 2 to 2048: They do show different performance, but did not inform more on SP.
DS GT=sadness R=fear
DV GT=sadness R=fear
GT GTnsadness R=fear
GT=sadness R=fear
Scenario
Exp1
Exp2
Expy
DS
16.08%
13.18%
52.09
OV
13.05%
13.18%
36.50
DS
0V
AV GT=happiness R=fear
AO
GT=happiness R=fear
GT=happiness R=fear
GT=happiness R=fear
AV
13.17%
13.18%
36.50
A0
12.83%
13.18%
12.17
Table 3: Emotion recognition accuracy results.
Figure 7: (Top) vagueness in audio reconstruction, (bottom) weak reconstruction difficult images.
The results against Exp; and Exp2 show a similar trend as with the shape data set. The accuracy with SP becomes here lower than uniformly random choice (for classes, random is about 14.29%)
7

---

Under review as a conference paper at ICLR 2021
We have also calculated accuracy using only the central and last frame of each example clip. The performance remains similarly low, so we only report in the table the numbers from the same conditions as the shape experiment, where accuracy is measured on all frames.
We have also confirmed the output from SP were "decent" reproductions of the raw input, as shown in fig. 7 (optimizing regularization Loshchilov & Hutter (2017) and introducing residuals He et al. (2015) did not improve performance significantly). However the DS numbers are the baseline and upper limit performance for the classifiers, which we believe low due to the difference between the original model's data set and RAVDESS. Other scenarios should intuitively be lower, which provides informal validity for our experiments, as well as motivates the next experiment.
Expy is simpler than Exp1 and Exp2. but trained on small subset of RAVDESS exclusively (one video frame per second, less than 0.02% of the data set). This specialization usually invalid in creating a strong classifier. The purpose here is to see the impact of SP, and opening perspective on joint training. Table the absolute performance low as a classifier, yet the benefit and cost of SP are coherent with shape classification. Also, the performance in AO falls at the same level as Exp; and Exp2. It indicates this SP implementation unable to reconstruct images well from audio only, which we expect from data significantly more complex than the shapes.
The diagonal blocks of SP's traverse are reported with a 0.00 mean, values ranging from -0.93 to 1.36. Here too the traverse model is validated posteriori, starting from random values. The range of values is also larger than the shape scenario, which we believe related to the more complex data set.
5 DISCUSSION & CONCLUSION
The shape classification experiment shows the effect of the sensoriplexer on sensory resilience. The following experiments on emotion classification show potential of the approach in a range of scenarios, despite currently poor results, partially suffering from the limited generalization power of the selected classifiers, and partially for the simple matrix-basee model for SP's traverse. The emotion recognition task may lack training data, or require different training features (e.g. generating components from the audio cepstrum, also referred to as the MFC mel-frequency components).
SP is also trained independently on the target data set, as an auto-encoder. generic emotion classifiers shows a disconnect between SP trained on RAVDESS, and their models trained on FER. Although the two datasets share most of their classes (7 out of 8), the nature of the data varies significantly. RAVDESS provides both higher quality and lower diversity, its restricted and well-calibrated environment. We expect joint training of SP, together with its downstream systems to converge to higher overall performance.
The experiment with
SP also provides mean to build agents resilient to sensory failures. We believe this is useful in itself, yet we expect a future generation of the concept to deal with agent modifications. A typical example is to "upgrade" sensor component on an active agent. In practice, different sensors have different nominal ranges, even in for the same model (sometimes due to heterogeneous manufacturing practices). SP in this paper could be affected under such change, much as seen in combining RAVDESS-traiee
SP and FER-trained classifiers. There is an opportunity for SP to dynamically learn to adapt to a new-yet semantically compatible-senoor.
Synesthesia is reduced in this work to the core meaning of "sensory mixture", which we model as a joint distribution over random variables representing sensory inputs. This decision serves the purpose for an agent to learn how to deal with sensory failures. Synesthesia is however a more elaborated mechanism Cytowic (2018). We expect new insights for artificial agents, from its study in Neurophysiology and Neuroanatomy. Besides, the biological neural correlates are not well understood yet. SP is inspired by synesthesia, rather than an attempt modeling it. As the understanding of the correlates advances, SP is likely to evolve or disappear, especially if more evidence of tight coupling between perception and cognition are confirmed.
The sensoriplexer neural component is practical mechanism for engineering more resilient artificial agents. The experiments in this paper validate the approach under two concrete scenarios, as well as show the potential for applications, such as robotics. The simple architecture selected for presenting SP can be seen as a baseline to explore more capable models, including the introduction of variational or recurrent variants. We will be exploring several of these paths in future work.
8

---

Under review as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
The authors would like to thank XXX for reviews on earlier drafts.
REFERENCES
Paul Bach-y Rita. Brain mechanisms in sensory substitution. Academic Press, New York, 1972.
Horace Barlow, Possible principles underlying the transformations of sensory messages. Sensory Communication, 1961, doi: 100555//mmppes777722222222220000
Camegie Mellon University CMU. CMUCam. http::/cmucam.ce///////// 2020. Accessed: 2020-06-19.
Richard E. Cytowic. Synesthesia. The MIT Press Essential Knowledge series, 2018.
Kaiming He, Xiangyu Zhang. Shaoqine Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385,, 2015. |RL/ttp:: http:/ org./bs////a/.oooooo aataasss/11... 1512. 03385
Geoffrey E. Hinton and Richard S. Zemel. Autoencoders, minimum description length and helmholtz free energy. In Proceedings of the 6th International Conference Neural Information Processing Systems, NIPS'93, 993, pp. 3-10, San Francisco, CA, USA, 1993. Morgan Kaufmann Puhlishers Inc.
Uta Jürgens and Danko Nikolić. Ideaesthesia: Conceptual processes assign similar colours similar shapes. Translational Neuroscience, 3, 03 2012. doi: 10.2478/s13380- 012-0000-1--0----
Steven R. Livingstone and Frank A. Russo. The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), April 2018. URL hhttt:: //opi. org/10........ooo...ooo..o...... 5281/211222 2et.://eoo..11111 1188976 Funding Information Natural Sciences and Engineering Research Council of Canada: 2012-341583
Hear the world
rescarch chair in music and emotional speech from Phonak.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in Adam. CoRR, abs/1711.051011 2017.
10. http:tttpp//axxx...rrrggo
00101]
Danko Nikolić. Practopoicsis: Or how life fosters a mind. Journal of theoretical biology, 373, 02 2014, doi:
Scott D. Novich and David M. Eagleman. Using space and time to encode vibrotactile information: Toward an estimate of the skin's 's achievable throughput. Experimental Brain Research, 233(10)27777-2788 June 2015. URL
doi: 10.1000../0 1007/s00221-015-4446
Tomas Pfister and Peter Robinson. Speech emotion classification and public speaking skill assessment. In Albert Ali Salah, Theo Gevers, Nicu Sehe, and Alessandro Vinciarelli (eds.), Human Behavior Understanding, pp. 151-162, Berlin, Heidelherg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-14715-99
Shangeth Rajaa. Facial emotion recognition for PyTorch and ONNX. https://githinn com/ shangeth/ 2019. Accessed: 2020-04-01.
Fero P Simoncelli. Vision and the statistics of the visual environment. Current Opinion in Neurobiology, 13
(2):144-149, April 2003. doi: (0.10166:00559-44888))))00000000 URL [ttpp/ttts///8...orr 80959 11959-43885 4388 (D3) D0047-3
Sony. Intelligent vision sensors with Al processing functionality, official press release of the 500 sensor. https:: www. sony. nett/ http://wwwwoooooooo/./////////// 06-19.
2020. Accessed: 2020-
Wikipedia. Sensor fusion Wikipedia, the free e httpp://....iippoaca..oooo///// Sensor fusion 2020. Accessed: 2020-06-19.
Jie Wu. Facial expression recognition with PyTorch.
Paci al Express on hom-RRoggniiio wtt.:niiiiio...oooiiiioe 2018. Accessed: 2020--4401.


Reviews:
[{"review_id": "alY5qxYMKRL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea and research direction itself is definitely interesting and worthy of pursuit. However, the execution is really poor. In addition to many improvements to clarity and writing, the proposed method is not at all novel and various variants for reconstructing one sensory modality from others have been proposed in the past:\n\nGu et al. \"Improving domain adaptation translation with domain invariant and specific information\" .arXiv [Preprint].arXiv:1904.03879, (2019).\n\nMurez et al.“Image to image translation for domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Salt Lake City, UT),4500–4509, (2018).\n\nLuo et al. (2018). “ViTac: feature sharing between vision and tactile sensing for cloth texture recognition,” in2018IEEE International Conference on Robotics and Automation (ICRA) (Brisbane,QLD: IEEE), 2722–2727.\n\nLee et al. (2019). “Touching to see”and “seeing to feel”: Robotic cross-modal sensory data generation for visual-tactile perception,” in 2019 International Conference on Robotics and Automation (ICRA) (Montreal, QC), 4276–4282.\n\nTatiya et al. \"A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization.\" Frontiers in Robotics and AI 7 (2020): 137.\n\nThese papers should be considered by the authors, discussed in related work, and used as a basis to propose something new in future work as there are still problems in that area that need solving. \n\nFinally, the notation is way more complicated than it needs to be for a simple encoder-decoder architecture. My advise to the authors is to simplify it. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZHQpAqaqukB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a neural component architecture, named Sensoriplexer (SP), with an aim to introduce resilience to agents performing machine learning tasks. Specifically, if different types of inputs can inform the decision of an agent, the SP aims to learn the dependencies among these inputs to make the agent's decision resilient to the absence of a subset of inputs. \n\nPros: In theory, the concept of SP is appealing and can have many practical applications. The paper does a decent job motivating the problem and elucidating different aspects of the SP framework with the help of examples. \n\nCons: \n1. Readability and Presentation: Section 3 of the paper, which lays the context for the application of SP, is not clear. The mathematical notations (such as of the form $(E_i)_i, d_E, etc.$) are not adequately defined or discussed. Similarly, the notation $M$ is used to define two separate quantities, which adds to the confusion. \n\n2. Significance of Results: The major issue that I find with the paper is the significance of empirical results. While the notion of adding resilience to ML architectures is the main motivation, the empirical results do not seem to provide convincing evidence for the utility of SP architecture in achieving it. The only result of note in the context seems to be on the visual shape classification task in the first set of experiments, where the 'A0' scenario achieves a better than chance performance (32.60% vs 25%) by the introduction of SP architecture. I am curious if this improvement in performance was enabled by the bias of the ML architecture to classifying one or two specific classes with better accuracy. \n\nThe message given by the experiments for the emotion recognition task is unclear to me. The addition of SP architecture does not appear to impact the performance of the three architectures evaluated. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5dbVz7t-5vw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n+ve Code shared. Power to reproducible science :)\n\nI think a reference to stacked auto-encoders is in order in addition to the reference on auto-encoders. Thank you for citing OG content though! You can also cite a range of papers for redundancy reduction including Bell &Sejnowski, Olshausen & Field\n\nI think the idea is more in line with sensory fusion than synesthesia. \n\nI don’t think I follow what you mean by traverse. It is stated in an obvious fashion but I don’t seem to get the reference. \n\nIt would be of value to compare the work against works such as joint audio-video speaker identification/localization. It seems like that would be a fair task to apply against. There are many models for this sort of work out there. \n\nThe major concern I have with this work is that while there is a nice narrative on sensory fusion but the model itself is a pretty vanilla stacked auto-encoder with fixed length of inputs. One obvious thing we know about the various sensors in the human system is that their sampling rates are different. For e.g. the time it takes for a photoreceptor to build potential again is different from say a hair cell in the ear. Modeling that perhaps might make the work more actionable. \n\nIt could be that I missed some core points in the paper but I don’t get the results in Table 2. Why is a drop in performance an impact of Sensoriplexer and why is this a good thing? Don’t you want to show that you are able to leverage information across modalities and do better, if so should not AV do better? \n\n“Also, the performance in A0 falls at the same level as Exp1and Exp2. It indicates this SP implementation is unable to reconstruct images well from audio only, which we expect from data significantly more complex than the shapes.” — I don’t understand this, I thought the idea was to classifiy emotions. \n\nOverall the paper has some nice ideas but I feel both the models and experiments need a bit more work. Feel free to convince me otherwise :) ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend rejection, based on multiple (mostly shared) concerns. While the authors address the concerns in their rebuttal, the unanimously negative scores remain. I don't see basis to accept the paper.
Decision: Reject