Venue: NeurIPS 2020 Workshop SVRHM

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ## Summary of the paper

This article proposes immediate fusion of data modalities for multimodal representation learning. The authors argue that the brain performs multimodal integration very early in the sensory processing to motivate a similar solution for artificial systems. To explore this idea, the paper presents the results of training a bimodal architecture, a convolutional LSTM, on a bimodal data set that combines MNIST with Free Spoken Digits. From their results, the authors conclude that immediate multimodal fusion provides better performance and robustness than late integration. 

## Summary of merits and concerns

### Merits

+ The topic is relevant and is likely to receive increased attention in the near future, especially if more challenging multimodal tasks can be appropriately designed for experimentation.
+ The paper is very easy to read, well-written, the results are presented very neatly in the figures.
+ Besides the results, the paper introduces an interesting network architecture that combines visual and audio inputs flexibly, as well as a simple multimodal data set, that may be used in future research.

### Concerns

- Since the empirical results are obtained on a newly constructed data set, not very challenging and carefully controlled, the results have intrinsically limited impact.
- For some of the contributions to be useful for the community, the code and data should be made available. Will they be?
- After reading the paper, there a few things for which I need further clarification, which I detail below.

## Evaluation and justification

I assess this submission very positively. While it is not the most important aspect to take into account, the presentation is great: the paper is easy to read, the introduction nicely motivates the problem, the review of related work is brief but relevant (perhaps more recent papers on multimodal representation learning should be reviewed), the experimental setup is fairly clear and the figures showing the results are quite neat. Beyond this, the topic is relevant for both the machine learning community and the neuroscience community, and the paper draws relevant connections between the two worlds. Furthermore, although of limited impact due to the nature of the data and the tasks, the results and conclusions are promising. I believe that this paper can be of interest and inspiring for the SVRHM audience and others and therefore I strongly recommend the acceptance for the workshop.

## Questions

I would like to have a couple of questions clarified:

- In Figure 1 and others, why is the performance worse with higher signal to noise ratio? This seems counter-intuitive.
- Why are not results of training with the original SNR presented in the paper?

I may be missing something, but these observations make be think that there may be some issues with the data set and the task that generate some artifacts like these.

## Minor comments and potential typos identified

I would like to note below some minor comments, suggestions and potential typos that the authors may take into account for future versions of the paper:

- Typo: "However, we now know that in many species, including humans, _that_ multisensory convergence occurs much earlier": extra _that_
- Suggestion: propose and consistently use a terminology to refer to each model (immediate, second layer, fully connected, etc.)
- Suggestion: in the figures, mark graphically the audio and visual SNR used for training, for example with a vertical and horizontal line.
- The performance of the unimodal visual model is near to perfect, as it is well-known from the extensive literature on the MNIST data set. However, the performance is far from perfect on the Free Spoken Digits data set. It would be desirable to construct a model that would solve the audio task as well as the visual task for easier comparisons.

review_title: A pleasant read with neat experiments and promising results

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper focuses on the demonstration that early fusion of data is preferable compared to later staged fusion. 
The study focuses on an interesting aspect of sensor fusion, whether it should be early in processing or later.

There seem to be a few major limitations with the experiment construction of the work. If the goal is to compare early versus late fusion, a starting point point could be to concatenate input representations together before a network, and then for other runs, fuse at higher/later stages of the network after allowing some pre processing. It is unclear to this reviewer why the authors chose to adopt a more complex strategy using C-LSTMs. Further, given that the more simpler experiments have not been explored, the experiments seem limited in scope and applicability to more common network architectures.

That said, the paper offers results for robustness of noise and on a simple easy to investigate dataset. This reviewer encourages the authors to continue this direction of work however consider more simpler hypothesis to test first and perhaps a few more challenging datasets to test the limits of their approach.
review_title: Interesting problem and motivation, however experiments have limited scope

final_score: rating: 7: Good paper, accept
review_content: This paper investigates at what stage of processing it is best to fuse multimodal input. It therefore generates a toy dataset combining written (MNIST) and spoken (Free Spoken Digit) data to mimic multimodal input.
It then compares different levels in artificial neural networks to merge the input and measures the resulting performance. Since the task without noise would be too easy they add noise to the datasets to make the task harder.

My first concern would be the data at hand. The visual and audio stimulus don't actually have any relation besides the class label. That leads to an unnatural setting and is even as a toy dataset not really reflecting the actual challenge. Also adding noise to the images renders it a multimodal denoising task which might not be the same as a setting where both image and audio are naturally hard (and not via noise).

A second concern would be with regards to capacity. I would like to ruled capacity issues out for this analysis. It would be interesting to see an analysis with a deeper and more overparametrized network to see at what stage it is optimal to merge the input signals.

Overall I think the paper provides careful evaluation and is not overstating its results and should therefore be accepted.

47: join -> joint (typo?)
review_title: Nice computational support for early fusion

Decision: Accept (Poster)