AutoMO-Mixer: An automated multi-objective multi-layer perspecton Mixer model for medical image based diagnosis
Authors: Xi Chen
Jiahuan Lv
Xuanqing Mou
Zhiguo Zhou
Venue: ICLR 2022 Submitted (2022)

Abstract: Medical image based diagnosis is one of the most challenging things which is vital to human life. Accurately identifying the patient's status through medical images plays an important role in treatment of diseases. Deep learning has achieved great success in medical image analysis. Particularly, Convolutional neural network CNN) can obtain promising performance by learning the features in a supervised way. However, since there are too many parameters to train, CNN always requires a large scale dataset to feed, while it is very difficult to collect the required amount of patient images for a particular clinical problem. Recently, MLP-Mixer (Mixer) which is developed based multiple layer perceptron (MLP) was proposed, in which the number of training parameters is greatly decreased by removing convolutions in the architecture, while it can achieve the similar performance with CNN. Furthermore, obtaining the balanced outcome between sensitivity and specificity is of great importance in patient's status identification. As such, a new automated multi-objective Mixer (AutoMO-Mixer) model was developed in this study.  In AutoMO-Mixer, sensitivity and specificity were considered as the objective functions simultaneously to train the model and a Pareto-optimal Mixer model set can be obtained in the training stage. Additionally, since there are several hyperparameters to train, the Bayesian optimization was introduced. To obtain a more reliable results in testing stage, the final output was obtained by fusing the output probabilities of Pareto optimal models through  the evidence reasoning (ER) approach. The experimental study demonstrated that AutoMO-Mixer can obtain better performance compared with Mixer and CNN.

Under review as a conference paper at ICLR 2022
AUTOMO-MIXER AN AUTOMATED MULTI- OBJECTIVE MULTI-LAYER PERSPECTON MIXER MODEL FOR MEDICAL IMAGE BASED DIAGNOSIS
Anonymous authors Paper under double-blind review
ABSTRACT
Medical image based diagnosis is one of the most challenging tasks which crit- icala to human life. Accurately identifying the patient's status through medical images play an important role in treatment. Deep learning has achieved great suc- cess in medical image analysis. Particularly, Convolutional neural network(CNN) can obtain outstanding performance by learning the features in a supervised way. However, there are too many parameters to train, CNN always requires large scale dataset to feed, while it is very difficult to collect the required amount of images for a particular clinical problem. Recently, MLP-Mixer (Mixer) which is developed based multiple layer perceptron (MLP) was proposed, in which the number of training parameters is greatly decreased by removing convolutional layers in the architecture, while it can achieve the similar performance with CNN. Furthermore, obtaining the balanced outcome between sensitivity and specificity is of great importance in patient's status identification. As such, new automated multi-ohjective Mixer (AutoMO-Mixer) model was developed in this study. AutoMO-Minn sensitivity and specificity were considered the objective fune- tions simultaneously to train the model and a Pareto-optimal Mixer model set can be obtained in the training stage. Additionally, since there several hyperpa- rameters to train, the Bayesian optimization was introduced. To obtain a more reliable results, the final output was achieved by fusing the output probabilities of Pareto optimal models through the evidence reasoning (ER) approach. The ex- perimental study on public datasets demonstrated that AutoMO-Mixer obtain better performance compared with Mixer and CNN.
INTRODUCTION
With the development of modern medicine, medical image has become essential way to carry personalized and accurate diagnosis. Due to the outstanding image analysis ability, deep learning has been widely used in medical image based diagnosis and has achieved great success(Zhang & An 2017 (hen et al. 2017) in the past years.
As there are so many parameters to train in current deep learning model such as convolutional neural network (Chua 1998), a large scale dataset is always required to feed the model. In particular, very
the parameters convolutional layer accounted majority of the parameters. However, difficult to collect sufficient medical image samples for particular clinical problem in practice. Recently, new deep learning model termed MLP-Mixer (Mixer)YTolstikhin et al.] 202T) which was developed based on multi-layer perception (MLP) was proposed. Compared with CNN, the convolutional layer removed from Mixer, leading to decreasing the architecture parameter scale sharply and Mixer can be trained sufficiently through small scale dataset. On the other hand, Mixer can achieve similar performance with CONNTolstikhin et al. 202I). In summary, Mixer is a better architecture for building medical image based diagnostic model.
Furthermore, building the balanced model between sensitivity and specificity is necessary al., 20088) in clinical diagnosis. Sensitivity indicates the proportion of people who are actually sick that are diagnosed as sick, whilst specificity indicates the proportion of people who truly healthy who are diagnosed healthy(Reitsma et al., 2005). As shown in
et al.

---

Under review as a conference paper at ICLR 2022
Table 1: An example of predicting distribution
True-abnormal
True-normal
Predicted-abnorre
35
Predicted-norma
55
Table 1 although the accuracy is high, sensitivity and specificity imbalance, resulting in higher rate of missed diagnosis. Therefore, a more balanced model is necessary and a multi-objective model which considers sensitivity and specificity simultaneously is needed.
As such, a new automated multi-objective Mixer (AuxeMO-Mixer) model developed this study. In AutoMO-Mixer, both sensitivity and specificity are considered the objective functions simul- tancously and a Pareto-optimal model set can be obtained through the multi-objective optimiza- tion(Zhou et al. (017)) in training stage. In addition, since there several hyperparameters which may affect the model performance AutoMO-Mixer, Bayesian optimization(Pelik) to train the hyperparameters. In testing stage, the Pareto-optimal models with balanced sensitivity and specificity are chosen so as to improve model diversity and stability. To obtain more reliable result, evidential reasoning (ER) Yang & Xu 2002) approach is used fuse the output of selected Pareto-optimal models to obtain final outcome. The experimental studies on two public medical im- age datasets demonstrated that AutoMO-Mixx can outperform Mixer and CNN, and more balanced results can be achieved as well.
(0055) is used
METHOD
2.1 FRAMEWORK
The framework of AutoMO-Mixer is shown in figure [ which consists of training stage and test- ing stage. In training stage, medical images are fed into multiple Mixer models with randomly initialization. Then sensitivity and specificity are considered as the multi-objective functions si- multaneously, and The Pareto-optimal Mixer model set can obtained through multi-objective optimization. Meanwhile, since there are several hyperparameters which may affect the model per- formance, Bayesian optimization is used to optimize the hyperparameters, In testing stage, models with balanced sensitivity and specificity are fused using the ER approach to obtain more reliable final prediction model.
Pareto-optimal model
M
multi-objective oplimization
M
M2
M,
Bayesian optimization
ER
Figure 1: The framework of AutoMO-Mixer model.
2

---

Under review as a conference paper at ICLR 2022
2.2 TRAINING STAGE
Assume {ai, ..., 0rk) }o represents the hyperparameters, where K is the number of hyperparame- ter and [3 denotes the parameters in Mixer models. M {m ma} represents the Mixer model, where q is the number of models.
The MLP-Mixer network is visual-oriented, all-MLP architecture as shown figure 2 consists consisting of four P Tolstikhin et al. 2021): splitting a picture into multiple patches shaped as a "patchesxchannels" table an imput; converting all patches into feature embedding with set of fully connected networks; refining feature information through N mixer layers; and classifying them with full-connectto layer after global average pooling. Mixer is divided into miken (feature refining along the columns direction of the table, i.e. spatial locations) and channel-mixer (feature refining along rows direction of the table, i.e. channels), all using MLP for feature extraction, each MLP consists of two layers of full-connected and an activation function GELU.
MLP
Fully-connected
Fully-connected
GELU
token-mixer (channeles X MLP)
channel-miie (patchels X MLP)
Global AveragePooling
(N-1) X Mixer Layer
Fully-connedtes
Fully colneccd
class
Mixer Layer
patches Xchannek channels
Figure The structure of Mixer.
Since there are two groups' parameters to train, the training stage is a bilevel iterative optimization process. other words, Mixers' model parameters and hyperparameters in AutoMO-Mixer are updated iteratively. The training stage is shown in figure 3 In the beginning, all the parameters are initialized randomly.
When the hyperparameters are determined, the Mixers' model parameters are trained first. To obtain balanced model, sensitivity and specificity are considered as objective functions simultaneously. they are:
TP
Ssen
TP + FN TN
Jspc
(2)
TN FP
where TP is true positive, TN is true negative, FP is false positive, FN is false negative. The goal is maximize feens fape simultaneously. that is:
mwx(faaa fape)
(3)
To optimize the above function, an iterative multi-objecteo immune algorithm (Iouu) al.
2017)(Gong et al.) 2008) is used. IMIA consists of six steps: initialization, cloning, mutation, deletion, update, and termination. First, the initial solution set denoted by D(t) (M1..., Mx)
3

---

Under review as a conference paper at ICLR 2022
Multi-objective optimization
Pareto-optimal model set
D0
C()
M0
D
M())
DE
f maxe Saxc Fae)
AUIC based
cione
imuutaee
unite
deletee
non-dominated sorting
sorting
2iee (DF (1) N
terations
1-AUC
inttaaiize
(1)
< ieraatims
m-nUU-
post-test distribution
probabilistic surrogate model
MP 1-AUC
accquisition function
Bayesian optimization payeeiaanoptimlzation
Figure 3: The illustration of training stage.
N. .... (m....., m 1,2,..,, will be replicated using the proportional cloning method. In the third step, probability of mutation is randomly generated for each model, and the model performs mutation when its probability is larger than the mutation probability. After the mutation, the new models are generated. If the models have same sensitivity and specificity, only one model re- mained and the all the other models are removed. Then the model size is kept through AUC based non-dominated sorting strategy. This training process will not stop until the maximum number of iterations is reached.
is generated, where M, {ma1.... ..., maghi i
Then the models with higher M;
Then the Pareto-oue Pareto-optimal Mixer model set is generated, where the model set size J. Now the optimal parameters for Mixer are obtained and hyperparameters are optimized. Since there are hyperparam- eters Pareto-optimal model selection and fusion phase, the weight for each Pareto-optimal model should be estimated, which is denoted by wy. As the balanced model between sensitivity and speci- ficity is desired, the ratio between them considered, which is faem less than 0.5 or greater than 1, the model considered as extreme imbalance, setting wy as O. The expression of wy is as follows:
Jepe or fape
Jeen When the ratio
X
Saen
(1 A)AUC, when 0.5 fiom <
fipe
W
Pipe
(1 A)AUC, when 0.5
Size
S 1, 1,,,...,J,
(4)
flon
flom
Other situations
where À indicates the importance of balance, and 1-A indicates the importance of AUC. After caleu- lating the Wj for each model, the weights are normalized, that is:
wy = 1,0 wy 1
(5)
Then the Pareto-optimal models are fused through ER approach. Assume there are models in Pareto-optimal model set, and the output probability (normal and abnormal) for each model is de- noted by in P3 = (P). PP).j [+].j=1, 1, 2,., where Ptt is the output probability of abnormal, and p2 is the output probability of normal. The final output probability Pr 1, 2 is obtained through the ER fusion strategy (Wang et al. 2006), that is:
Pf F.(P5 wy),i 1,,,, ..., J, 1,2
4

---

Under review as a conference paper at ICLR 2022
where ER is:
12
III 5=1
(wjPy +1 ( + wy) ) 17 (1 (1-w;)]
fin
, 1,2
- x 11 (1- w;]]
==1
The normalized factor is:
2
wy Pj +
wy) - II (1- (1-ww)1
(8)
c1
j=1
The hyperparameter a can be optimized through Bayesian optimization berg et 2016), where the objective function is:
2005 Springen
fH min - AUC)
(9)
Eqs. (3) and (9) are optimized iteratively until reaches the termination critation.
2.3 TESTING STAGE
An illustration of testing stage is shown in figure 4 The medical images are fed into the trained Pareto-optimal Mixer models and the output probabilities from each model can be obtained. Then ER approach is used to fuse the outputs all the Mixer models and final output can be obtained.
Pareto-optimal Mixer model set
M
ER
Pr ER(P .1,))
Pp > p/
normal
Way
j= 12
p/ s Pin
abnormal
M 100
#) pp+
Figure 4: The illustration of testing stage.
3 EXPERIMENTS
3.1 MATERIALS AND SETUP
The two public databases used in this paper are MRI images brain tumors(Panigran) 2021) and CT images of brain tumors. It contains MRI scans the brain, and each type of brain has 1500 images. We selected 900 images for modeling. The next dataset is collected from COVID19-BEB 19-reletd papers from medRxiv, bioRxiv, NEJM, JAMA, Lancet, etc. contains 349 CT scans that are positive for COVID-19, This dataset can be used perform classification and automatically detect COVID-19 on CT scans. The distribution of the two types of data is shown Table 2. The examples of two datasets shown in figure 5
2020). The first dataset aims to help people for detecting
Before feeding the model, all the images are resized into 224 x 224 images, and each image is divided into 14 14 patches, where each patch size 16 16. There are hyperparameters,
5

---

Under review as a conference paper at ICLR 2022
Table 2: Distribution of two types of data
Train
Train
Test
Test
Abnormal
Normal
Abnormal
normal
Brain Tumor
360
360
90
90
COVID-19
280
318
69
79
brain (abnormal)
COVID-19 (abnormal)
brain (normal)
COVID-19 (normal)
Figure 5: The examples two datasets.
MP and A, which the range is set between 0 and 1. The network structure contains five parameters, they are: the number of Mixer layers, the number of channels(Hiddee size C), the number hidden layer neurons of Moken-mixin((MLL) dimension Ds), and the number of hidden layer neurons of mixnnel- dimension Dc). The settings are shown table
Table 3: The range of values for Mixer network structure parameters
parameters
range of values
Number of layers
[2, 3, 4]
Hidden size 256*[1, 1.2, 1.4, 1.6]
MLP dimension Ds
196* 3, 4,
MLP dimension De
256 4, 6, 8, 10]
In comparative study, Mixer and CNN model are evaluated. For fairness of comparison, the number of layers of the two models is required to be equal. We make the number of layers five. The four parameters of the Mixer network are set to 5, 256. 392, 1024, respectively.
AUCoohao et al. 2011) is ideal indicator to evaluate the reliability of the prediction results, and the larger the AUC shows that the better the classification of the model. Accuracy (ACC) represents the correct rate of diagnosis and is also important indicator for evaluating the model. Therefore, sensitivity (SEN), specificity (SPE), AUC, ACC are used for evaluation. All the experiments are performed five times and the average results are shown.

---

Under review as a conference paper at ICLR 2022
3.2 RESULTS
The evaluation results on two datasets are shown table 4 5 and figure 6 seen that Mixer can achieve the similar performance, while the number of parameters to train is greatly less than CNN. Furthermore, AutoMO-Mixer outperforms both Mixer and CNN all four evaluation results, which means that AutoMO-Mixer is more promising model.
respectively. It can be
Table 4: The results on Brain Tumor dataset
SEN
SPE
AUC
ACC
Mixer
0.92
0.92
0.96
0.92
CNN
0.92
0.91
0.95
0.92
AutoMO--Mixr
0.93
0.93
0.97
0.93
Table 5: The results on COVID-19 dataset
SEN
SPE
AUC
ACC
Mixer
0.75
0.72
0.80
0.73
CNN
0.71
0.70
0.77
0.70
AutoMO-Mixer
0.83
0.80
0.88
0.81
Brain_Tumor
COVID-19
SEN
SEN
SPE
SPE
AUC
ACC
ACC
0.88
0.9
0.92
0.94
0.96
0.98
0.4
0.8
Mixer
CNN
AutoMO-Mixer
Mixer
CNN
AutoMO-Mixer
Figure 6: The performance of three models.
CONCLUSIONS
In this study, new model termed AutoMO-Mixer was developed for image based diagnosis. In AuxeMO-Mixer sensitivity and specificity are considered as the objective functions simultaneously and a Pareto-optimal Mixer model set can obtained. Meanwhile, Bayesian optimization used to train the hyperparameters to improve model performance. In addition, obtain more reliable results, the Pareto-optimal models with a balance between sensitivity and specificity are selected and the ER approach is used to fuse their output for results. The experimental results two public medical image datasets showed that AutoMO- can outperform Mixer and CNN. Furthermore, more datasets will be used validate model performance.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Priyanka Banerjee, Frederic O. Dehnbostel, and Robert Preissner. Prediction is a balancing act: Importance sampling methods to balance sensitivity and specificity of predictive models based on imbalanced chemical data sets. Frontiers in Chemistry, 6:362, 2018.
I. O. Chua. CNN: A Paradigm for Complexity. CNN: A Paradigm for Complexity, 1998.
Maoguo Gong, Licheng Jiao, Haifeng Du, and Liefeng Bo. Multiobjecttive immune algorithm with nondominated neighbor-basd selection. Evolutionary computatioon 16(2):225 255, 2008
LuisBlanche, Covid-19 lung ct scans.
httpss//www..aallle.cmm///uuubb
covidct 2020.
Maciej Mazurowski, Piotr Habas, Jacek Zurada, Joseph Lo, Jay A Baker, and Georgia Tourassi. Training neural network classifiers for medical decision making: The effects of imbal- anced datasets on classification performance. Neural networks, 21(2-3):477-336, 2008.
Abhranta Panigrahi. Brain tuttr-tttttootoionnnnooo mri.
brrain-umor-ddteeee on-mri 2021.
M. Pelikan. Bayesian Optimization Algorithm. Hierarchical Bayesian Optimization Algorithm, 2005.
J. B. Reitsma, A. S. Glas, Aws Rutjes, Rjpm Scholten, P. M. Bossuyt, and A. H. Zwinderman. Bivari- ate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology, 58(10);982-990, 2005.
D. Shen, G. Wu, and H. I. Suk. Deep learning in medical image analysis. Annual Review of Biomedical Engineering, 19(1):221- 2-222,8.88 2017.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems, 29: 4134 4142, 2016.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy, Mlp-mixer: An all-mlp architecture for vision, 2021.
Ying-Ming Wang, Jian-Bo Yang, and Dong-Ling Xu. Environmental impact assessment using the evidential reasoning approach. European Journal of Operational Research, 174(3):1885-199 2006.
Jian-Bo Yang and Dong-Ling Xu. On the evidential reasoning algorithm for multiple attribute de- cision analysis under uncertainty. IEEE Transactions on Systems, Man, and Cybernetics-Poen A: Systems and Humans, 32(3):289-344, 2002.
Y. N. Zhang and M. Q. An. Deep learning-ant and transfer learning-basee super resolution reconstruc- tion from single medical image. Journal of Healthcare 1ngimmerrin.20077222777720)01100 2017.
Peilin Zhao, Steven CH Hoi, Rong Jin, and Tianbo YANG. Online auc maximization. 2011.
Z. Zhou, M. Folkert, P. Iyengar, K. Westover, and W. Jing. Multi-objective radiomics model for predicting distant failure in lung sbrt. Physics Medicine & Biology, 62(11):4460, 2017.
8


Reviews:
[{"review_id": "r7TJ7Iq1mHp", "reviewer": "Reviewer_Bur6", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my view, this paper makes several technical and conceptual errors, and fails to make any noteworthy contributions. The paper is difficult to read and skips over many important details necessary to understand and reproduce the work. It also contains numerous typos and grammatical errors, including a misspelling of “perceptron” in the title. \n\nMOTIVATION\nIn the introduction, an argument is made to justify the choice of MLP-Mixer for small-scale datasets, in which it is stated “the convolutional layer is removed from Mixer, leading to decreasing the architecture parameter scale sharply and Mixer can be trained sufficiently through small scale dataset.” I don’t believe this to be true - to my knowledge it has not been shown that MLP mixers are more efficient in terms of number of parameters than CNNs, in fact research on transformers and MLP-Mixers shown that the lack of inductive bias in these models make them less efficient than CNNs. They only benefit when model capacity is large and the dataset is very large. In the original MLP-Mixer paper it was necessary to pre-train on ImageNet21k and Google’s proprietary JFT-300M dataset with 300M images to make MLP-Mixer competitive. The whole rationale for selecting MLP-Mixer for this task seems incorrect.\n\nRELATED WORK\nThere is no related work section. The paper fails to communicate how this work relates to the current state of the art in medical image classification, other works using MLP-Mixers, or Bayesian Optimization for hyperparameter tuning.\n\nMy next concern is about the goal of balancing sensitivity and specificity with Pareto-optimal models. It is not clear to me why a balance of sensitivity and specificity is indeed a good thing in general, and it has not been convincingly argued by the authors. I can think of counter-examples in medicine, e.g. in breast cancer screening different countries set different goals of recall rates because they value FP and FN differently. Why should we try to balance these if they do not have the same cost to society?\n\nTECHNICAL DESCRIPTION & CONTRIBUTIONS\nThe description of the approach (AutoMO-Mixer) does not provide enough detail to understand or reproduce the work. The overall scheme depicted in Figure 1 is confusing. It is stated that this figure depicts a training and test stage, but this is not evident from the figure. The main content of the methodology is a recap of the MLP-Mixer architecture, along with a depiction of it in Figure 2. On page 3 it is stated that there are “2 groups” to train, but it is not clear what these two groups are. I am assuming that these are two groups of MLP-Mixers, one being the Pareto-optimal set? The “Multi-objective” goal which gives the model it’s name is really a single objective - namely to optimize for a balanced model defined in Eq. 3. This is achieved through an evolutionary algorithm which operates on a group of MLP-Mixer models, a strange choice. It would make more sense to build this objective into the training of the MLP-Mixer itself. The authors also claim some novelty by using a Bayesian optimization of hyperparameters but this is also problematic - first, they are using an off-the-shelf BayesOpt solution and secondly the hyperparameters they are optimizing for are just the hyperparameters of the Pareto-optimal model selection. It has nothing to do with training the MLP-Mixers.\n\nEXPERIMENTS\nThe experimental results are also concerning. To start with, the experiments are conducted on extremely small datasets from Kaggle - a set of 3000 MRI images and a set of 700 Covid CT scans. The MLP-MIxers were trained with random initialization - when dealing with such small datasets the standard procedure should be to use transfer learning. The architecture used in not one of the standard specifications defined in (Tolstikhin et al, 2021). A comparison is made against stand-alone MLP-Mixers and CNNs, and to keep the comparison “fair” the number of layers is chosen to be the same. The correct setup would be to choose similar #parameters or FLOPS. The experiments report the sensitivity and specificity along with accuracy, but never do an ablation to show how performance is affected by class imbalance which is very relevant for sensitivity and specificity.\n\nCLARITY & ERRORS\nFinally, there are many instances where explanations are unclear or terms are misused. For example, Table 3 purports to list the MLP-Mixer parameters, but they have in fact listed the hyperparameters. The parameters of the model are of course the tunable weights. They hyperparameters of the multi-objective optimization which are optimized for are states as MP and \\lambda, but it is never stated what MP is. Figure 3 and Figure 1 are very difficult to understand even after reading the paper.", "summary_of_the_paper": "This work proposes AutoMO-Mixer, an approach for combining multiple MLP-Mixer models for medical image classification. In short, the paper builds on recent work in MLP-Mixer models, describing an ensemble learning scheme designed to balance sensitivity and specificity.", "summary_of_the_review": "The motivation for this work appears to be flawed, the technical description is unclear and lacks novelty, there are flaws in the experimental design, and the paper contains several errors and lacks clarity.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "oRcgyv5oA-", "reviewer": "Reviewer_SKkc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nI cannot find a significant strength from this study.\n\n# Weakness:\n1. No novelty. Everything looks a direct application of existing technique.\n2. Poorly motivated. Why MLP Mixer should be tried for medical images? Why multi-objective optim should be introduced here? Even if the experiments show the \"proposed\" model looks better, so what? I cannot find any specific design for medical images.\n3. Lack of empirical results. Only \"CNN\", \"Mixer\" and the so-called \"AutoMO-Mixer\" is compared, why? There are so many variants in every part...\n4. Bad presentation. Writing and figures needs improvement. No story in this paper.", "summary_of_the_paper": "This study is a simple combination of multiple-objective optimization and MLP Mixer on medical image application, with poor experiments to support any conclusion. Everything looks a direct application of existing technique. I don't understand why this study is submitted to a top conference.", "summary_of_the_review": "A simple combination of existing techniques, with poor experiments to support any conclusion. No story.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "SkhqZxtLAT", "reviewer": "Reviewer_rxpX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Obtaining sufficient annotated medical data for training data-hungry neural networks is currently a challenge. The paper proposed an alternative to CNNs, MLP-Mixer, which requires less training data.\n2. Usually, medical datasets are not balanced, which leads to an imbalance in sensitivity and specificity. The authors propose to use a multi-objective immune algorithm to balance between sensitivity and specificity. \n3. Results seem promising (but comparisons to some other related approaches might be missing).\n4. Using Evidential Reasoning (ER) to fuse model set is interesting (but lacks comparison to simple voting-based fusion strategies).\n\nWeaknesses: \n1. “In summary, Mixer is a better architecture for building medical image-based diagnostic model.” - The authors make a very strong claim without providing any reference or evidence to back up the claim. \n\n2. The technical novelty of the paper is limited. The proposed method is the fusion of MLP-Mixer, Bayesian optimization, and ER. However, application in the medical domain is important. \n\n3. The notation is confusing, and the method section is hard to follow. ‘N’ denotes the number of layers of the second paragraph in 2.2 and is used later for D(t). Does N represent the number of layers of models or the number of models in the initial set D(t)? \n\n4. M represents the Mixer model, which is a set consisting of ‘q’ models. It’s hard to follow what M represents. \n\n 5. It’s is not evident that the better performance of AutoMO-Mixer is because of multi-objective function or better hyper-parameters with Bayesian optimization.\n\n6. The use of Evidence Reasoning (ER) to fuse the probabilities of different models is not well justified. \n\n7. Literature review and background information about EV and IMIA are missing. \n\n8. Figure 2 is taken/inspired from the original MLP-Mixer paper. Authors should mention this in the caption with referencing.\n\nMinor Points:\n1. “Then the models with higher M_i.” It is not clear what does high means in this context. \n\n2. I would suggest changing the caption of the figures to make it self-explanatory. Authors should add a gist of the figure or a few lines about the information being visualized or conveyed. \n\n3. The authors may want to consider moving background information about MLP-Mixer in section 2.2 (Training stage) to a new section (literature review/background). It breaks the flow while understanding the training strategy of the proposed algorithm. \n\n4. There are typos in the paper (and in the title!). \n\n\nSuggestions for improvements:\n\n1. Imbalance in sensitivity and specificity is mainly due to the imbalance dataset. Thus, it’s essential to compare the performance with training with classes weighted according to the number of samples in the training set and Focal loss [1] to justify using the multi-objective function and multi-objective immune algorithm. \n\n2. Authors should define ‘Pareto-optimal’ as the audience might not be familiar with the terminology.\n\n3. Authors should compare it with other fusing strategies (averaging, voting, etc.) to justify the use of Evidence Reasoning (ER) to fuse the probabilities of different models.\n\n4. In Table 4, I would also suggest fine-tuning hyperparameters of the CNN and Mixer model with Bayesian optimization to empirically demonstrate that the better performance of AutoMO-Mixer is because of multi-objective function, and not because of better hyper-parameters with Bayesian optimization.\n\n5. “In summary, Mixer is a better architecture for building medical image-based diagnostic model.”  Though Mixer-MLP has fewer parameters than CNN, they don’t need to be the best of medical images. I suggest authors run experiments on a large-scale medical dataset such as The Cancer Genome Atlas (TCGA) [2] to back up such a strong claim.\n\n[1] Lin TY, Goyal P, Girshick R, He K, Dollár P. Focal loss for dense object detection. InProceedings of the IEEE international conference on computer vision 2017 (pp. 2980-2988).\n\n[2] Hutter C, Zenklusen JC. The cancer genome atlas: creating lasting value beyond its data. Cell. 2018 Apr 5;173(2):283-5.\n", "summary_of_the_paper": "The authors proposed a new method for learning imbalanced medical image datasets. They propose to use the recently propose MLP-Mixer instead of CNN  to reduce the number of trainable parameters and thus can be trained with fewer images as compared to CNNs.\nTo handle the class imbalance in the datasets, they use multi-objective loss functions to balance between sensitivity and specificity. Bayesian optimation is used for finding the optimal hyperparameters. The proposed method is evaluated on two medical image datasets, showing improvements over CNN and Mixer baselines.\n", "summary_of_the_review": "While the work shows promise, particularly in the medical imaging domain, more experiments and comparisons with existing methods (as mentioned above) are required to improve the paper further and justify the use of EV and multi-objective function to handle the imbalance between sensitivity and specificity. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HifeiarkpTu", "reviewer": "Reviewer_5jFR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper are in recognizing the importance of targeted solutions towards smaller data sets, through introducing and evaluating a convolutional layer-free neural network architecture. The attempted balancing of the final sensitivity and specificity metrics by weighing individual Mixer models using an iterative multi-objective immune (genetic) IMIA algorithm, is also relatively novel.\nHowever, the paper also has a number of concerning issues:\n1. The standard method for balancing sensitivity/specificity with deep learning models, is to the best of our knowledge, selecting the corresponding operating threshold/point on the model output on a held-out validation dataset. It might be clarified as to whether this was performed in evaluating the (single) Mixer and CNN models for the results as presented in Tables 4 and 5, especially since the CNN sensitivity/specificity values appear approximately balanced.\n2. While the goal for the Mixer models is stated to be maximizing f_sen and f_spe simultaneously, the max(.) function used in Equation 3 appears to have the default definition of taking the maximum value amongst the two values, which does not appear the intended meaning. The equation might thus be clarified if appropriate, e.g. f = min(abs(f_sen – f_spe)) if balancing their values is the priority.\n3. For IMIA, it is stated that models with higher M_i = {m_i1 … m_iq} will be replicated. The definition of m_iq might be clarified; are they model performance by AUC, and if so, was some training data further held out for internal validation of such metrics?\n4. It is claimed that the Mixer model set is Pareto-optimal (Page 4). This Pareto-optimal property might be more explicitly proven.\n5. The Mixer model set size is stated to be J (Page 4). The value and selection process for J might be described, for the experiments attempted.\n6. MP is stated to be one of the two hyperparameters for IMIA together with lambda, however MP does not appear to be introduced or reference anywhere else in the paper. Is MP actually J?\n7. The network structure is then stated to consist of five parameters. However, only four parameters are then described, and listed in Table 3.\n8. The fairness of the comparison between (single) Mixer and CNN models was stated to be imposed by the number of layers being equal in both models (Page 6). However, the structure and training details of the CNN models are not indicated. It is therefore difficult to determine as to whether the CNN architecture used was reasonable, and whether the potential of the CNN models had been maximized.\n9. The individual contributions (i.e. Mixer architecture, IMIA, ER fusion) were not disentangled through proper ablation experiments. As such, it is difficult to determine the main source of the performance improvement for AutoMO-Mixer, and indeed whether some of these contributions were necessary. For example, there appears no reason why CNN models could not be combined by IMIA, or even through basic ensembling, for comparison purposes.\n10. The results are stated to be the average of five independent runs (Page 6). The standard deviation/confidence interval attained might thus be considered to be presented.\n11. There are a number of spelling & grammatical issues, e.g. “perspecton” in the title, “developed based (on) multiple layer perceptron” in the abstract, “accounted (for) a majority” in the introduction section, etc.\n\n", "summary_of_the_paper": "This paper proposes an AutoMO-Mixer framework for medical image diagnosis, consisting of several features: Firstly, the use of the MLP-Mixer deep learning model architecture, which is stated to achieve reduced parameter scale via the elimination of convolutional layers, thus making the architecture more suited to the training of small-scale datasets. Secondly, the learning of a (Pareto-optimal) Mixer model set, by weighing the multiple objectives of achieving a balanced sensitivity/specificity, and overall performance as measured by AUC. Thirdly, the fusion of the outputs of these selected models in the Mixer model set, by an ER fusion strategy. The AutoMO-Mixer framework was compared against a single Mixer alone and a CNN with equivalent number of layers, on two publicly-available CT images datasets on brain tumor and COVID-19 detection respectively. While all three approaches performed similarly on the brain tumor classification dataset, AutoMO-Mixer was shown to perform appreciably better than the (single) Mixer and CNN approaches, on the COVID-19 dataset.\n", "summary_of_the_review": "While the motivations of the paper are well-founded, there remain multiple significant concerns, in particular over the methodological details and the comprehensiveness of the evaluation.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers recommended a rejection. The authors of the paper did not respond.
Decision: Reject