Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 

+ve Code shared. Power to reproducible science :)

I think a reference to stacked auto-encoders is in order in addition to the reference on auto-encoders. Thank you for citing OG content though! You can also cite a range of papers for redundancy reduction including Bell &Sejnowski, Olshausen & Field

I think the idea is more in line with sensory fusion than synesthesia. 

I don’t think I follow what you mean by traverse. It is stated in an obvious fashion but I don’t seem to get the reference. 

It would be of value to compare the work against works such as joint audio-video speaker identification/localization. It seems like that would be a fair task to apply against. There are many models for this sort of work out there. 

The major concern I have with this work is that while there is a nice narrative on sensory fusion but the model itself is a pretty vanilla stacked auto-encoder with fixed length of inputs. One obvious thing we know about the various sensors in the human system is that their sampling rates are different. For e.g. the time it takes for a photoreceptor to build potential again is different from say a hair cell in the ear. Modeling that perhaps might make the work more actionable. 

It could be that I missed some core points in the paper but I don’t get the results in Table 2. Why is a drop in performance an impact of Sensoriplexer and why is this a good thing? Don’t you want to show that you are able to leverage information across modalities and do better, if so should not AV do better? 

“Also, the performance in A0 falls at the same level as Exp1and Exp2. It indicates this SP implementation is unable to reconstruct images well from audio only, which we expect from data significantly more complex than the shapes.” — I don’t understand this, I thought the idea was to classifiy emotions. 

Overall the paper has some nice ideas but I feel both the models and experiments need a bit more work. Feel free to convince me otherwise :) 
review_title: A model to emulate sensory resilience/fusion using stacked autoencoders. The experiments use audio and video to do shape and emotion classification.  

final_score: rating: 2: Strong rejection
review_content: The idea and research direction itself is definitely interesting and worthy of pursuit. However, the execution is really poor. In addition to many improvements to clarity and writing, the proposed method is not at all novel and various variants for reconstructing one sensory modality from others have been proposed in the past:

Gu et al. "Improving domain adaptation translation with domain invariant and specific information" .arXiv [Preprint].arXiv:1904.03879, (2019).

Murez et al.“Image to image translation for domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Salt Lake City, UT),4500–4509, (2018).

Luo et al. (2018). “ViTac: feature sharing between vision and tactile sensing for cloth texture recognition,” in2018IEEE International Conference on Robotics and Automation (ICRA) (Brisbane,QLD: IEEE), 2722–2727.

Lee et al. (2019). “Touching to see”and “seeing to feel”: Robotic cross-modal sensory data generation for visual-tactile perception,” in 2019 International Conference on Robotics and Automation (ICRA) (Montreal, QC), 4276–4282.

Tatiya et al. "A Framework for Sensorimotor Cross-Perception and Cross-Behavior Knowledge Transfer for Object Categorization." Frontiers in Robotics and AI 7 (2020): 137.

These papers should be considered by the authors, discussed in related work, and used as a basis to propose something new in future work as there are still problems in that area that need solving. 

Finally, the notation is way more complicated than it needs to be for a simple encoder-decoder architecture. My advise to the authors is to simplify it. 
review_title: Not novel enough and poor presentation

final_score: rating: 3: Clear rejection
review_content: The paper proposes a neural component architecture, named Sensoriplexer (SP), with an aim to introduce resilience to agents performing machine learning tasks. Specifically, if different types of inputs can inform the decision of an agent, the SP aims to learn the dependencies among these inputs to make the agent's decision resilient to the absence of a subset of inputs. 

Pros: In theory, the concept of SP is appealing and can have many practical applications. The paper does a decent job motivating the problem and elucidating different aspects of the SP framework with the help of examples. 

Cons: 
1. Readability and Presentation: Section 3 of the paper, which lays the context for the application of SP, is not clear. The mathematical notations (such as of the form $(E_i)_i, d_E, etc.$) are not adequately defined or discussed. Similarly, the notation $M$ is used to define two separate quantities, which adds to the confusion. 

2. Significance of Results: The major issue that I find with the paper is the significance of empirical results. While the notion of adding resilience to ML architectures is the main motivation, the empirical results do not seem to provide convincing evidence for the utility of SP architecture in achieving it. The only result of note in the context seems to be on the visual shape classification task in the first set of experiments, where the 'A0' scenario achieves a better than chance performance (32.60% vs 25%) by the introduction of SP architecture. I am curious if this improvement in performance was enabled by the bias of the ML architecture to classifying one or two specific classes with better accuracy. 

The message given by the experiments for the emotion recognition task is unclear to me. The addition of SP architecture does not appear to impact the performance of the three architectures evaluated. 
review_title: Major issues with the presentation and significance of empirical results

Meta-review: All three reviewers recommend rejection, based on multiple (mostly shared) concerns. While the authors address the concerns in their rebuttal, the unanimously negative scores remain. I don't see basis to accept the paper.
Decision: Reject