Suppression helps: Lateral Inhibition-inspired Convolutional Neural Network for Image Classification
Authors: Chengyuan Zhuang
Xiaohui Yuan
XUAN GUO
Venue: Submitted to ICLR 2023 (2023)
Keywords: Lateral Inhibition
Convolutional Neural Networks

Abstract: Convolutional neural networks (CNNs) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field. For better recognition, the dimensions of depth and width have been explored, leading to convolutional neural networks with more layers and more channels. In addition to these factors, neurobiology also suggests the widely existing lateral inhibition (e.g., Mach band effect), which increases the contrast of nearby neuron excitation in the lateral direction, to help recognition. However, such an important mechanism has not been well explored in modern convolutional neural networks. In this paper, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) design. Our naive design incorporates the low-pass filter, while eliminating the central weight to mimic the inhibition strength decay. The inhibition value is computed from the filtering result of the input, with a simple learnable weight parameter per channel for multiplication to decide the strength. Then the inhibition value is subtracted from the input as suppression, which could increase the contrast to help recognition. We also suggest an alternative using depthwise convolution, as a general form. Our design could work on both the plain convolution and the convolutional block with residual connection, while being compatible with existing modules. Without any channel attention along the channel dimension, the preliminary results demonstrate an absolute improvement of 3.68\% and 0.69\% over AlexNet and ResNet-18, respectively, in the ImageNet data set, with little increase in parameters, indicating the merits of our design to help feature learning for image classification.

Under review as a conference paper at ICLR 2023
LATERAL INHIBITION-INPPIRER STRUCTURE FOR CONVOLUTIONAL NEURAL NETWORK ON IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks (CNNs) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field. For better recognition, the dimension of both depth and width has been explored, leading to convolutional neural networks with more layers and channels. In addi- tion to these factors. neurobiology suggests lateral inhibition (lateral antagonism, e.g. Mach band effect), a widely existing phenomenon for vision that increases the contrast and sharpness of nearby neuron excitation in the lateral direction help recognition. However, such mechanism has not been well explored in the design of convolutional neural network. In this paper, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) structure. Our naive design uses the low-pass filter mimic the strength decay of lateral interaction from neighbors regarding the distance. One learnable parameter per channel is applied to set the amplitude of the low-pass filter by multiplication, which is flexible to model various lateral interactions (including lateral inhibition). The convolution result is then subtracted from the input, which could increase the contrast and sharpness for better recognition. Furthermore, learnable scaling factor and shift are applied to adjust the value after subtraction. Our lateral cuhibition-inspieee the convolutional block with residual connection, while being compatible with the existing modules. Preliminary results demonstrate obvious improvements on the ImageNet dataset for AlexNet (7.58%) and ResNet-18 (0.81 %), respectively, with little increase in parameters, indicating the effectiveness of our brain-similar de- sign to help feature learning for image classification from different perspective.
(LI) structure works both plain convolution and
INTRODUCTION
In recent years, convolutional neural networks (CNNs) (Hinton et al.) 2012 Simonyan & Zisser- man 2015) Szegedy et al.) 2015 He et al.] (016) have become powerful and popular tools since deep learning emerged for image classification in the computer vision field. They have recorded accord-breaking performance and outperformed traditional methods (Quinlan] 1986 Cortes & Vap-l
nik 1995) with hand-crafted features (Lowe) 1999 Dalal Triges 2005) on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (Deng et al.) 2009 Today, convolutional neural networks still possess unique merits, as they have been studied the most, along with the fact that convolution has strong connection with the human vision system and image processing, making them good models for feature learning research.
Different factors have been explored to improve recognition performance of convolutional neural networks. VGGNet (Simonyan & Zisserman) (0155) applies a small convolution kernel size (3 x 3) for increased network depth, while ResNet (He et al. 2016) introduces deep residual learning to make training very deep deep networks feasible. The success of such networks indicates that depth is a crucial factor for recognition performance. Wide Residual Networks (Zagoruyko & Komodakis, 2016), on the other hand, demonstrate width as another important factor to improved performance. In addition these factors, neurobiology suggests the widely existing lateral inhibition (lateral antagonism, e.g. Mach band effect, shown in Fig. []. phenomenon that increases the contrast and sharpness of nearby neuron excitation in the lateral direction, also important to help feature learning.

---

Under review as a conference paper at ICLR 2023
Actual Intensity
Perceived Intensity
Illustration of Mach band effect. Due lateral inhibition, the actual perception is different from the real input, leading to increased contrast and sharpness along the boundary (see curves to- wards opposite direction), where the dark area becomes darker and the bright area becomes brighter.
Figure 1:
However, such an important mechanism has not been well explored in convolutional neural network design. In recent years, Yang et al. (2015) propose traditional method using manual double- opponent design (antagonism) with spatial sparseness constraint for boundary detection, following the color-opponent mechanism in the human vision system, to highlight the color boundary while suppressing texture edges. This double-opponent design is further extended (Gao et al., 2015) for color constancy. Cao et al. (2018) apply lateral inhibition mechanism in the inference stage of the pre-trained VGGNet (Simonyan & Zisserman 2015) for visual attention map and saliency detection. Hasani et al. (019) propose their surround modulation design, using a manually defined kernel from the difference of Gaussian (DoG) function, explore the feasibility of incorporating the lateral inhibition mechanism. This fixed design only applied to half channels of the initial layer (with better performance than applying to the full channels). It is still worth exploring flexible lateral inhibition-inspired design for convolutional neural network, to incorporate such mechanism and make more similar the brain in the training stage for image classification.
In this paper, inspired by the findings of neurobiology, we explicitly explore the filter dimension in the lateral direction and propose our lateral inhibition-inspired (LI) structure, which incorporates the lateral inhibition mechanism in a flexible manner inside modern convolutional neural networks. Our simple naive design applies the low-pass filter with the central weight eliminated, to mimic the decay of inhibition strength from neighbors regarding the distance. To set the amplitude of the low-pass filter by multiplication, learnable parameter per channel is used, for the flexibility of modeling various lateral interactions (including lateral inhibition). The convolution result is then subtracted from the input, which could increase the contrast and sharpness for better recognition. Furthermore, learnable scaling factor and shift are applied to adjust the value after subtraction. Our lateral inhibition-inspired (L.I) method works on both plain convolution and convolutional block with residual connection, while being compatible with the existing modules. Preliminary results of our simple naive design demonstrate obvious improvements on the ImageNet dataset for AlexNet (7.58%), R8 8 (0.81 81% 1)), respectively, with little parameter increase, indicating the effectiveness of our brain-similar to help feature learning for image classification from a different perspective.
Our main contributions can be summarized as:
We propose lateral cnhibition-inspirees structure for modern convolutional neural network design, which could participate in the training stage for image classification.
Our design is flexible compared to the fixed difference of Gaussian (DoG) filter, and could be applied to all layers (and channels) with competitive performance.
We are the few to explicitly model lateral inhibition as well as other lateral interaction to make the network more brain-similar, given the flexible weight design.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
Neurobiology. Lateral inhibition is common in sensory system. Mach reports the Mach bands effect RATLIFF (1965) due to lateral inhibition that the human vision system perceives increased contrast and sharpness along the boundary, where the dark area becomes darker and the bright area becomes brighter. Blakemore et al. (1700) discover that lateral inhibition plays an important role in orientation detection within the human vision system. It helps to remove redundancy for the existence of inhibitory signals, as large stimulus covering a whole receptive field would be relatively ineffective in excitation of neurons,
Traditional method. Yang et al. (2155) propose their double-opponent design with spatial sparse- ness constraint for boundary detection, by mimicking the color-opponent mechanism in the human vision system. The single opponent uses manual antagonism such red-green and blue-yellow, while the double opponent combines two opposite single opponent in either center-surround or side- by-side manner for increased contrast. This design highlights the color boundary while suppressing texture edges. The double-opponent design is further extended (Gao et al. 2015) for color con- stancy.
Convolutional Neural Network. Local Response Normalization (LRN) (Hinton et al. 2012) is introduced in Nlexlota and is not trainable. Normalization performed for the value each position (x, y) considering neighbors at the same position from nearby channels, with the denomi- nator as a sum of the squared neighbors, given a scaling factor, bias, as well as the exponential value for the denominator. It applies lateral inhibition across nearby channels via competition. Batch normalization (loffe & Szegedy, 2015) applies normalization within individual channels, using an mini-batch mean (shift) and variance, then a scaling factor and re-shift parameter, to constrain all values in a certain range. Layer normalization (Ba et al. 2016) only consider one single instance, but performs normalization across all channels using the average and variance from each channel. Group normalization (Wu & He, (018) further extends layer normalization by dividing channels into groups for consideration. Besides normalization methods, Cao et al. (2018) propose their lat- cral inhibition design using pretrained VGGNet, as top-down step for attention map and weakly supervised saliency detection. The inhibition value consists of average term from the neighbors within the same channel to protect high response zone, and a differential term only consider the difference from higher activation nearby for inhibition, with weight decay regarding the distance. The whole formula including term coefficients are intuitively decided and unlearnable, and only happens in inference stage. Hasani et al. (2019) propose surround modulation for lateral inhibition, to make network closer to the brain. It applies two gaussian filters (5x5) with 1:1 weight (fixed) from the difference of Gaussian (DoG) function. However, this fixed design lacks flexibility, which is only applied to half channels of the initial layer within a small convolutional neural network, due to the performance drop when applying to full channels or the following layer. The channelwise normalization (center pool) in Weighted normalization (Pan et al.) (021) improves LRN (Hinton) ctt al. 2012) from AlexNet by adding extra parameters such as a scaling factor and bias, while fur- ther adding surround pool for normalization within each channel does not help on ImageNet dataset. DivisiveNorm (iileer et al. (021) also similarly follows the design of LRN at each location along the channel dimension, but changes the numerator be squared, and adds weight scaling factor for each unnormalized activations inside the exponentially weighted sum. The hyper parameters are manually set for stable training.
METHOD
Our lateral iinibittioninsspiee (LI) structure explicitly incorporates the lateral inhibition mechanism into the design of convolutional neural network, to increase contrast and sharpness for more accurate perception. Unlike methods only involving in the post-processing stage with the original pre-trained model unchanged, our method participates in the training stage.
Our lateral inhibition-inspiredd inspired (LI) design takes two steps: first, it generates the lateral interaction value based on the input within the spatial neighborhood of each channel; then it subtracts that value from the original input. Details will be explained in the following parts.
3

---

Under review as a conference paper at ICLR 2023
3.1 LATERAL INTERACTION DESIGN
We design our naive lateral interaction based on neurobiological findings. follows the principle that the strength of interaction (including inhibition) from active neuron decreases as the distance to that neuron increases in the lateral direction. For implementation, we take the exsiting Gaussian low-pass filter with central weight eliminated mimic the strength decay from neighbors. Here, we denote the function of lateral interaction as L.I, Gaussian low-pass filter as G, the central weight value of the low-pass filter C, convolution operation as Conv, the learnable amplitude weights as W (with one parameter per channel for multiplication to separately decide the channelwise amplitude of Gaussian low-pass filter G), and the input features as x for our lateral cntibitiioninsppiree structure.
The weights of the 2D Gaussian low-pass filter (G) are calculated using Equation [ Here and denote the spatial coordinates regarding to the center of the Gaussian low-pass filter, and a denotes the standard deviation for weight decay speed from the center (we set filter size as and I):
G(u, v) 2TT²
20
The lateral interaction function given the input feature x at channel i is defined in Equation 2 First, convolution input feature using Gaussian low-pass filter G is applied, then the corresponding convolution result form the central weight value (of the low-pass filter G) is subtracted, to only have interaction from the neighbors. Finally, multiplication is performed by the amplitude parameter (from the learnable weights W) to decide the amplitude of the Gaussian low-pass filter G per channel. Different from Surround Modulation (Hasani al. 2019) which applies 1:1 weight using two Gaussian low-pass pass filters in the difference of Gaussian (DoG) function to model lateral inhibition, our learnable weights W more flexible (could be zero, positive or negative) to fit various lateral interactions (for none, lateral inhibition or even the opposite when necessary).
Li(x;) (Cono(G, C x.) W
3.2 APPLYING LATERAL INTERACTION
We denote the output function as O for applying lateral interaction on input feature x at channel i. Then the output function is defined in Equation 3
with lateral interaction result subtracted.
0 = -I((x))
(3)
3.3 ADDITIONAL PARAMETERS
Additional parameters are applied since the interaction result will be subtracted from the original feature map. Parameters include scaling factor (v), shift (m), and bias (b). Combination of these parameters can be explored to find the best setting.
0 = (O(x,) + +))++ +
(4)
3.4 STRUCTURE PLACE
Our lateral inhibition-inspee inspired (LI) structure works on both the plain convolution and the convolu- tional block with residual connection. For the plain convolution, it is placed after that convolution (and the activation function), as shown in Fig. 2 For the convolutional block with a residual con- nection, it is placed inside the residual block before the first convolution, as shown in Fig. 2
4 EXPERIMENTS
Dataset. We use the ImageNet- Ik data set (Russakovsky et al., 2015) to evaluate proposed methods. ImageNet- Ik dataset consists of 1k object categories, with 1.2 million images for training and 50k images for validation.
4

---

Under review as a conference paper at ICLR 2023
Lateral Inhibition
Convolution
Convolution
Convolution
Lateral Inhibition
Figure 2: Place for our lateral inhibition inhibition-inspired inspired (LD) structure. It is after the plain convolution or inside the residual block before the first convolution, as shown on the left right side, respectively.
Experiment Settings. We use the Pytorch (Paszke et al. 2019) deep learning framework and follow the official training example. We adopt the standard settings: image crop size of 224 x 224, initial learning of 0.01 for AlexNet (Hinton et al. (012) (which decays every 30 epochs by factor of 10), momentum of 0.9, and weight decay of 0.000T. We use standard data augmentation, which includes random horizontal flipping, random scale and ratio changes in the range of [0.08, 1.0] and [0.75, 1.33], respectively. Models are trained with batch size of 128 and batch accumulation of 2. Experiments are performed on GPUs, with 90 epochs for AlexNet following MMlllr et al. (2021) (more epochs show little difference) and 100 epochs for ResNet following common practice.
Results. Table shows the results of the baseline model AlexNet (Hinton al.) 2012), related methods, as well as our naive lateral inhibition-inspired listed methods improve the baseline significantly, our method outperforms others by a large margin. Layer normalization only consider one instance, which fits better for dynamic input rather than batch input. Group normalization has further division into groups, therefore limits its performance. Weighted normalization and Divisive normalization are similar in the normalization design, except the power number the numerator and the denominator, as well as the weighted sum. However, most of them lack the consideration of neighbors within the same channel for lateral interaction, which our method does.
inspired (LI) design on ImageNet dataset. While all
Table shows the results of the baseline model ResNet (He al. 2016) and naive lateral inhibition inspired (I.I) design on ImageNet dataset. Although ResNet contains batch normalization to improve performance, our design makes the improvement further by 0.81%.
Design Choice. From Table 3 we can see that, with a single scaling factor, our design already outperforms other methods. Among all the design variants, scaling first with bias is better than shift first then scaling. While they are both much better than only scaling, the difference between the best two are not large.
5 CONCLUSION
In this paper, we propose our lateral inhibition-inspired structure for convolutional neural network to make more brain-similar on image classification task. We explicitly explore neurobiology find- ings, using the guassian low-pass filter with central weight eliminated to mimic the strength decay regarding the distance. With a learnable weight, our deign is flexible to model lateral inhibition as well as other lateral interaction. Our design works on both plain convolution and the convolutional block with residual connection, while being compatible with the existing modules. Preliminary re- sults using AlexNet and ResNet demonstrate obvious improvements on the ImageNet dataset for
5

---

Under review as a conference paper at ICLR 2023
Table 1: Top-1 accuracy of AlexNet models (with only plain convolution) ImageNet dataset. LI
denotes our lateral inhibition-inspired (LI) design, with little increase in parameters and GFLOPs. Surround Modulation is added after every convolutional layer as other methods in the experiments.
Method
Top-I Acc (%)
Params.
GFLOPs
Improv. (abs)
AlexNet (baseline)
52.82
62.38M
1,137
+GroupNorm (Wu & He
2018)
57.76
62.38M
1.137
4.94
+oayerNorr Ba al.
2016)
58.81
63.68M
1.137
5.99
+BatchNorm lotte & Szegedy
2015
59.04
62.38M
1.138
6.22
WeightedNorm--C (Pan al. 2021
59.41
62.38M
1.137
6.59
+ DivisiveNorm (Miller et al.
2021
59.54
62.38M
1.137
6.72
+LI (ours)
60.39
62.38M
1.143
7.58
Table Top-1 accuracy ResNet18 models (contains residual block) on ImageNet dataset. LI
denotes our lateral inhibition-inspired (LJ) design, with little increase in parameters and GFLOPs. Our structure is applied to both the initial plain convolution and the following residual blocks. Please note that ResNet already contains batch normalization to increase performance.
Method
Top-1 Acc (%)
Params.
GFLOPs
Improv, (abs)
ResNet18 (baseline)
70.30
11.69M 11.69M
1.822
+LI (ours)
71.11
1.838
0.81
image classification, with little increase in parameters. We hope that our research could inspire researchers to reconsider the value of this track for better feature learning.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint
arXiv: 1607.00000 2016.
Colin Blakemore, Roger H S Carpenter, and Mark A Georgeson. Lateral inhibition between orien- tation detectors the human visual system. Nature, 228(5266):37-39) 1970.
Chunshui Cao, Yongzhen Huang, Zilei Wang, Liang Wang. Ninglong Xu, and Tieniu Tan. Lateral inhibition-inspee insooieedd convolutional neural network for visual attention and saliency detection. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
Navneet Dalal and Bill Triggs. Histograms oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and patlern recognition (CVPR105), volume 1, pp. 886-893. IEFE, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Chao-Bing Gao, Kai-Fu Yang, Chao-Yi Yi Li. and Jie Li. Color constancy using double- opponency. IEEE transactions on pattern analysis and machine intelligence, 1973-,997331 2015.

---

Under review as a conference paper at ICLR 2023
Table 3: Top-1 accuracy of AlexNet models (with only plain convolution) ImageNet dataset. L.I denotes our lateral cebbitioo-inspired (LI) design. Shift, scaling, and re-shift means that our design variants are changed by adding additional parameters.
Method
Top-1 Acc (%)
Params.
GFLOPs
Improv. (abs)
AlexNet (baseline)
52.82
62.38M
1.137
+LI (scale)
59.87
62.38M
1.143
7.05
+LI (shift-scaling)
60.26
62.38M
1.143
7.44
+LI (scale-bias)
60.39
62.38M
1.143
7.58
Hosein Hasani, Mahdich Soleymani, and Hamid Aghajan. Surround modulation: bio-inspired connectivity structure for convolutional neural networks. Advances in neural information pro- cessing systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey E Hinton, Alex Krizhevsky, and Ilya Sutskever. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1 106-1166 1114 2012.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of Machine Learning Research, volume 37, pp- 448-456, 2015.
David G Lowe. Object recognition from local scale- inveriant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pp. 1150-1157. leee, 1999.
Michelle Miller, Yeon Chung, and Kenneth D Miller. Divisive Feature Normalization Improves Image Recognition Performance in AlexNet. In International Conference Learning Represen- tations, 2021.
X Pan, E Kartal, L S Giraldo, and O Schwartz. Brain-inspired weighted normalization for CNN image classification. International Conference Learning Representations (ICLR), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and Others. PyToorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems, 32:8026-8037, 2019.
J Ross Quinlan. Induction of decision trees. Machine learning, 1(1):81-106, 1986.
F RATLIFF. Mach bands: Quantitative studies on neural networks in the retina. Holden- Day (Holden-Day series in psychology). 1965.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and Others. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 252, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Proceedings of 3rd International Conference on Learning Representationss 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018

---

Under review as a conference paper at ICLR 2023
Kai-Fu Yang, Shao-Bine Gao, Ce-Feng Guo, Chao- Yi Li, and Yong-Jie Li. Boundary detection ing doublc-opponencc and spatial sparseness constraint. IEEE Transactions on Image Processing, 256552286--578, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016.
8


Reviews:
[{"review_id": "uM31Bm53z8", "reviewer": "Reviewer_GVUz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nSummary:\nThis article starts from the perspective of neuroscience, hoping to design a neural network that conforms to human cognition. The original intention of the article is good, but there is nothing special about the method of the article. Regarding the writing, methodology, and experiments, this article is far below the acceptance criteria for a good conference like ICLR. Reviewers put a lot of effort into reviewing each article, so hopefully, authors will write each article well before submitting it.\n\n", "summary_of_the_review": "\nSee \"Summary Of The Paper.\" Regarding the writing, methodology, and experiments, this article is far below the acceptance criteria for a good conference like ICLR.", "strength_and_weaknesses": "\n(Positive) Although this article is very poor, its exploratory spirit, specifically the exploratory spirit of exploring neural networks that conform to human cognition, is worthy of recognition.\n\n\n(Negative) The authors claim some gain in their approach. But as we all know, after introducing attention or dynamic mechanism, a general neural network can have some improvement. In particular, the worse the network, the more obvious the improvement.\n\n(Negative) In fact, filtering is not a new thing in neural networks.\n\n\n(Negative) A very working method should be validated on large-scale tasks. In addition to classification, there is detection, segmentation, and so on.\n\n\n(Negative) The writing of this article is very poor. There are multiple copies of the text in the article. For example, there are multiple repetitions of text in the abstract and introduction. A good article should have a better way of expressing it.\n\n\n(Negative) Gaussian filtering and bilateral filtering are not new in neural networks. The method proposed in the article is equivalent to a bilateral filtering non-local neural network.\n\n(Negative) The article associates its method with depth-wise convolution. This is very imprecise and arrogant. If this is true, is MLP-Mixer also a special case of this article?\n\n(Negative) As I said before, adding attention or dynamics to the neural network, such as senet, sknet, acnet, dynamic nets, will bring gains. There are too many such examples. So the results in Table 2 are not dazzling at all.\n\n(Negative) The numbers in Equation 5 are so intuitive and natural.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ok-SHSEAckf", "reviewer": "Reviewer_edpQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper incorporates the lateral inhibition of neuron into the Deep network for image classification. Results show the effectiveness of the proposed mechanism. The lateral inhibition is not novel but the idea to explore the usability of the lateral inhibition into artificial neural network is interesting.  The key weakness of this paper is that there are no any explanations why their design improves the performance.  \n", "summary_of_the_review": "It can be seen that the work is not complete with only five pages as the main text. Many important issues such as explaining why the design of lateral inhibition improves the performance. Computationally, what’s the difference between channel attention and the inhibition computation?  In summary, the technique contribution is simple.  \n\nI encourage authors to enhance their work by addressing 1-2 key questions mentioned above, adding content and depth to the paper.\n\n", "strength_and_weaknesses": "The lateral inhibition based idea to explore the usability of the lateral inhibition into artificial neural network is interesting and useful. Such an approach should be strongly supported. \n\nThe key weakness of this paper is that there are no any explanations why their design improves the performance.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JhO4VvJYby9", "reviewer": "Reviewer_KkWZ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a type of neural networks that uses lateral inhibition inspired layers. Lateral inhibition in this paper refers to the effect where the contrast of nearby neuron excitation in the lateral direction is increased for recognition. Inspired by this, the paper proposes to use a Gaussian low-pass filter with a zero center weight along with a learnable channel weight to mimic this effect. A flexible alternative would be a depthwise convolution. Then this output is used to subtract from the input tensor to form the final output. This lateral inhibition inspired design is tested with AlexNet and ResNet18 for image classification on ImageNet, and shows improvements at little additional cost.", "summary_of_the_review": "This is a paper inspired by the lateral inhibition effect. The motivation is great, but the method is too simple, and the paper does not show the strength of the method through large-scale and comprehensive experiments. Overall, I think the paper is below the conference standard.", "strength_and_weaknesses": "This paper is motivated by a neurobiological effect named lateral inhibition. It is always fascinating to see methods that are inspired in this way because it may reveal some important angles that the academic community has overlooked.\n\nBut this paper seems to be less prepared for this conference. The method is interesting but very simple: it adds a couple of convolutional layers to existing architectures. The baseline architectures are not from recent works, and the model scales in the test are also small (e.g. ResNet18, AlexNet). I also didn’t find the visualizations mentioned in the paper. I hope that the authors could add more experiments to the paper, show larger experiment settings, and polish the paper a bit more.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8pd0pVqmwM", "reviewer": "Reviewer_Ct44", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to add a biologically inspired lateral inhibition mechanism into deep convolutional networks for image recognition. When incorporated into AlexNets and ResNets, LI seems to improve performance on ImageNet classification without increasing trainable parameters. The authors examine the LI filter weights and find a biologically-resemblant center-surround pattern of inhibition.", "summary_of_the_review": "The proposed work is preliminary and lacks the novelty and quality of work expected at ICLR. I do not recommend accepting this paper at this stage. I suggest the authors to please consider a suitable workshop for this work and a significant extension of this work could be suitable for the ICLR audience with wider evaluation using suitable baselines and tasks.", "strength_and_weaknesses": "Strengths:\n1) There are considerable gains on AlexNet and ResNet while using LI, however, I would like to note that these are preliminary results as also highlighted by the authors.\n\nWeaknesses:\n1) The proposed work lacks novelty, several methods in the past have tried to apply a very similar lateral or divisive inhibition mechanism to deep convolutional networks and have reported gains in image classification performance (particularly when added to AlexNet). See [1] [2] and [3] for example. The proposed work is almost exactly similar to [1].\n2) The evaluation is very preliminary and lacks comparison to suitable baselines (other kinds of normalization such as BatchNorm, LayerNorm, etc.) or other normalization techniques such as [1, 2] which are very relevant.\n\nReferences:\n1. Hasani, H., Soleymani, M., & Aghajan, H. (2019). Surround modulation: A bio-inspired connectivity structure for convolutional neural networks. Advances in neural information processing systems, 32.\n2. Miller, M., Chung, S., & Miller, K. D. (2021, September). Divisive Feature Normalization Improves Image Recognition Performance in AlexNet. In International Conference on Learning Representations.\n3. Pan, X., Giraldo, L. G. S., Kartal, E., & Schwartz, O. (2021). Brain-inspired weighted normalization for CNN image classification. bioRxiv.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject