ICNN: INPUT-CONDITIONED FEATURE REPRESENTATION LEARNING FOR TRANSFORMATION-INVARIANT NEURAL NETWORK
Authors: Suraj Tripathi
Chirag Singh
Abhay Kumar
Venue:  (2020)
Keywords: Transformation-invariance
Reconstruction
Run-time Convolution Filter generation

Abstract: We propose a novel framework, ICNN, which combines the input-conditioned filter generation module and a decoder based network to incorporate contextual information present in images into Convolutional Neural Networks (CNNs). In contrast to traditional CNNs, we do not employ the same set of learned convolution filters for all input image instances. And our proposed decoder network serves the purpose of reducing the transformation present in the input image by learning to construct a representative image of the input image class. Our proposed joint supervision of input-aware framework when combined with techniques inspired by Multi-instance learning and max-pooling, results in a transformation-invariant neural network. We investigated the performance of our proposed framework on three MNIST variations, which covers both rotation and scaling variance, and achieved 0.98% error on MNIST-rot-12k, 1.12% error on Half-rotated MNIST and 0.68% error on Scaling MNIST, which is significantly better than the state-of-the-art results. Our proposed model also showcased consistent improvement on the CIFAR dataset. We make use of visualization to further prove the effectiveness of our input-aware convolution filters. Our proposed convolution filter generation framework can also serve as a plugin for any CNN based architecture and enhance its modeling capacity.

Under review as a conference paper at ICLR 2020
ICNN: INPUT-CONDITIONED FEATURE REPRESENTA- TION LEARNING FOR TRANSFORMATION-INVAAIINY NEURAL NETWORK
Anonymous authors Paper under double-bling brind review
ABSTRACT
We propose a novel framework, ICNN, which combines the input-conditioned filter generation module and a decoder based network to incorporate contextual information present in images into Convolutional Neural Networks (CNNs). contrast to traditional CNNs, do not employ the same set of learned convolu- tion filters for all input image instances. And our proposed decoder network serves the purpose of reducing the transformation present in the input image by learning to construct a representative image of the input image class. Our proposed joint supervision of input-aware framework when combined with techniques inspired by Multi-instance learning and max-pooling, results in a transformation-invariie neural network. We investigated the performance of our proposed framework on three MNIST variations, which covers both rotation and scaling variance, and achieved 0.98% error on MNIST-rot- 12k, 1.12% егтоr on Half-rotated MNIST and 0.68% error on Scaling MNIST, which is significantly better than the state-of- the-art results. Our proposed model also showcased consistent improvement on the CIFAR dataset. We make use of visualization to further prove the effectiveness our input-aware convolution filters. Our proposed convolution filter generation framework can also serve plugin for any CNN based architecture and enhance its modeling capacity.
INTRODUCTION
Deep learning has been used to obtain promising results for various machine learning applications in recent times. Computer Vision has been at the frontier, driving its success with the introduction of Convolutional Neural Networks (CNNs). Face Recognition Li et al. (215)), Image Classification Deng et al. 2009). Action Recognition CChrron et al. 2015)) and Speech AnalysixAAbel- et al. (2014): Yenigalla et al. (2018) are some of the areas which use CNNs extensively.
LeNet, first introduced by Lecun et al. (1998) served as the prototype for the modern-day Convolu- tion Neural Network such as VGGNNe Simonyan & Zisserman (2014) and ResNet He et al. (201)). These networks first consist of several layers of convolution, activation, and subsampling layers. The convolution filters are learned to extract local features from the image patches, which are sub- sampled using either max-pooling or average-pooling. Pooling reduces feature map resolution and sensitivity of the output shifts and distortion. These local features are further combined hierarchi- cally to form high-level features. The high-level features are then fed to the fully connected layers form a compressed 1-D feature vector, which is then fed to a soft-max layer for Image Classi- fication. These network uses techniques like data augmentation Fawzi et al. (2016) and auxiliary classification Yin & Liu (018) to surpass many pre-established boundaries in Image Classification.
Although there has been significant improvement in image classification, local features extracted by learned filters keep information regarding neighboring pixels of the image patch, irrespective of its location in the image. This makes networks prone to errors when they encounter images having variations like rotation, scaling, etc., as this requires networks to withhold complete structural information of the image up until the soft-max layer. Designing neural networks robust to such variations have been under research recently, resulting in many techniques which help propagate structural information of the given image till classification.

---

Under review as a conference paper at ICLR 2020
Data augmentation Fawzi al. (2016) is widely used approach which artificially increases the training data by incorporating different affine transformations, adding noise, scale changes to the images present in the original dataset based prior knowledge. LLPpo et al. (2016) utilizes parallel Siamese architectures along with different transformation sets, inspired by multi- instance learning, and pooling operator, whereas MINTIN Zhang al. (2018) uses Input Normal- ization and combines it with Maxout operator to tackle these variations. Image features like, SIFT Lowe (2004) has been used extract features from key points, which are invariant to scale and rotation of the image.
generic framework has been proposed in this work for generating input-conditioned convolutional filters along with a decoder network which serves as generator module for learning representative images, which incorporates contextual information present in images into CNN models. distinc- tion with traditional CNNs, our framework generates unique set of input-aware filters based on the given image, and thus enables the network improve modeling capacity. We also employ a de- coder network which given the abstract features of an input image tries to construct a transformation free representative image of the input class.
We combined our proposed framework with techniques inspired by multi-instance learning Wu et al. (2013) and max mox-pooling operator Boureau et al. (2010) to generate transformation lutional neural network feature representations. Unlike data augmentation. commatiion-nvariantt invariant convo- model does not treat the original and its transformed instances as independent samples; we generate high-level repre- sentation from all the instances followed by max-pooling layer (element-wise maximum of all representations) which are then passed as an input to the further layers. Combination of our input- aware filters, Convolution layers, element-wise maximum over feature representations, and Decoder network makes the final representation independent from the variations.
Implementation of this network topology makes use of parallel Siamese networks which a com- bination of our proposed filter generator and decoder framework, convolution, pooling, and dense layers. We kept the convolution, pooling and dense layers consistent with those Laptev et al. (016): ZZang et al. (2018) to prove the effectiveness of our proposed input-aware framework. This topology supports weight sharing and takes a set of transformed image instances as input. We ana- lyzed the performance of our proposed Tranformationninvaaiia network variations of datasets and achieved the error rate of 0.98% on MNIST-rot- 12k Larochelle et al. (2007), 1. 1.12% on half- rotated MNIST Jaderberg et al. (2013) and 0.68% scaling MNIST which beats the state of the art accuracies as mentioned in Zhang et al. (2018). We also achieved consistent improvement on CIFAR dataset when compared with the baseline methods. Thus, validating the effectiveness of our proposed approach to handle transformations present in the images.
The following the main contributions of this work:
The first attempt, to the best of our knowledge, to make use of a decoder network to con- struct representative image of the input class in order to mitigate transformations present in the input image.
We also proposed run on-time generation of input-conditioned convolution filters in our model architecture.
The framework proposed in this paper can easily be integrated with any CNN based net- work to enhance its ability to incorporate contextual information present in images.
RELATED WORK
Recent research to acquire transformation invariance convolutional neural networks have led to four main techniques. This includes Data Augmentation, TIPOOLING, MINTIN, and usage of transformationninvaaaan features like SIFT. David G. Lowe proposed SIFT Lowe (2004) to extract distinctive invariant features present in images, which can be used for matching different object or scene views reliably. The method required selection of key points in the image and generation of features robust to local affine transformation by SIFT descriptors. However, designing these general general-purpose purpose cenfffrmmttionnnnvaiian features a tedious and expensive process, and they can only handle very specifie variations in the original data.
2

---

Under review as a conference paper at ICLR 2020
Data Augmentation increases the number of training samples in the original dataset by artificially introducing variations like rotations, scale changes, random crop, etc., based on prior knowledge. Although CNNs modeling capacity is flexible enough to support increase in data samples, it re- sults in increased training time and heavy usage of computational resources. The substantial increase in the number of parameters, necessary tackle variations, can make network prone to over or under-fitting.
TI-POOLING Laptev al. (2016) introduced transformation invariance in neural network using multiple instance learning (MIL) and pooling operator together. The proposed network consists of parallel Siamese network with transformed images as input. TI-POOLING operator then applied to do max- max-pooling over the transformations from the parallel network. The TI-POOLED transfor- mation invariant features are then fed soft-max layer for image classification.
MINTIN Zhang et al. (2018) uses an input normalization module, to precalibrate the input image using prior knowledge, in combination with Maxout operator which can be employed into any CNN based architecture to address variations like rotation and scaling.
3 APPROACH
The proposed architecture employs input-conditioned convolutional filters to extract features from an input image. Framework for feature extraction from input images consists of two modules: Input- conditioned Filter Generation, which generates a unique set filters given an input image, and Convolution Module, which applies the generated set of filters on the input extract context- dependent feature maps. This framework, Figure 2 is briefly explained below.
Before passing input images to the proposed feature extraction framework, make use of transfor- mations as described in Laptev et al. (016)). A predefined transformation set 4 is used to generate a collection of transformed instances of the given input image ((x), é E 4. Figure [ shows parallel instances of partial Siamese Networks consisting of our proposed feature extraction framework, two traditional convolutional (3x3 kernel size, kemnels = 80, 160 respectively) and pooling (2x2 kernel size, stride = 2) layers, with ReLU as the activation function, which take the generated set of trans- formed image instances as input to produce high-level feature representations. We kept the same hyperparameter settings as in Laptev et al.] (2016); /Zhag et al. (2018) to show the improvement in adding our proposed framework. Following the notation of Zhang et al. (2018), we also denote high-level feature vectors by fo(x), where x and 0 denote the given input and all the parameters of the proposed modules, kernels and weights respectively. Thus for n transformed input instances, will obtain feature vectors denoted by folo)(z)), feature vectors, we apply element-wise maximum on all fo(d)())), which makes the output (xr) either transformation invariant, in many cases, or least makes it less dependent the variations according to the Lemma defined in Wu et al. (2015) and Laptev & Buhmann 2015)
..., n. After extracting high-level
(o(x) max (x))
We hypothesized that the generated output representation (o(x) is an abstract depiction of the input image as takes the max of all the feature vectors generated from different variations of the image. Therefore, these features should have the sufficient information required to construct the image without any unnecessary transformation. Following this intuition, we introduced a convolution- based decoder network which takes this abstract representation as input and tries construct an image without any transformation.
Output representation (x) is also passed to dense layer, which makes of ReLU activation function and Dropout technique, followed by a final output layer with softmax activation function. This deep learning architecture employs the joint supervision, Lestal of the Cross-Entropy. Leg. and reconstruction loss, L2 - loss, and uses Adam optimizer. The À parameter is tuned as part of the training process.
Lraatt = LCE + X Treconstrructon crror
Siamese networks make use of weight sharing among parallel instances, which leads to a drastic reduction in memory usage. In other words, memory footprint of the network is not directly related to the number of parallel instances in the Siamese network.
3

---

Under review as a conference paper at ICLR 2020
Output Softmax Layer
Dense Layers
Proposed Convolution
Framework
Weighht sharing
Proposed Convolution Framework
Max-
pooling
Weight sharing
V
Decoder
Proposed Convolution Framework
Network
Input-conditioned Filter
Visual Context-aww aware Feature representations
Transformation- invariant Features
Image Reconstruction
Generator Convolution
Module
Figure 1: ICNN: Model Architecture
Input-conditioned filter generation: In theory, this generator module be implemented of any deep differentiable functions. However, due to the remarkable success of convolutional neural networks in computer vision related tasks Deng al. (2009): Simonyan & Zisserman (2014), decided to make use of convolution based operations to implement this module. Given an input image X, it is first passed spatial size of the generated feature map. This feature map is then passed to a Deconvolutional layer, which performs fractionally-strided convolutions [17], generate distinctive set of filters. To remain consistent with the architecture in Laptev et al. (2016); Zhang et al. (2018), we fix the kernel size and number of kernels of the input-aware filters be 3x3 and 40 respectively. We experimented with kernels of different sizes for the convolution and deconvolution layers work with the above described input-aware filters hyperparameters and selected 3x3 and 2x2 respectively. We hypothesized that the use of different kernel sizes in deconvolutional layers will result in the capture of different level features.
Convolution layer followed by a Max-pool layer to decrease the
Convolution module: A set of generated filters and image sample is passed as input to the convolution module. This module replicates the work of traditional convolutional layer and gen- erates encoded representation of the input image by convolving it with the given set of filters, followed by a pooling layer. No bias term is used this module. The proposed filter generator and convolution module, in basic terms, is generalized version of standard convolution layer that can be represented as our proposed framework by generating the same set of filters indepen- dent of the visual context present in the input image. Our input-aware filter generation framework focuses on encoding visual context information into the convolutional filters that in turn generates contexl-dependent feature maps when convolved with input image. In principle, the proposed filter generation framework can be easily combined with any CNN based architecture to incorporate visual context information.
The generated output representation (x()) is fed in to the convolution based decoder network. Specifically, the decoder network serves the purpose of generator and utilizes combination of deconvolution, pooling, and convolution layers to construct the representative image of the input image. The use of decoder network followed the intuition that by learning to construct transformation free image in parallel to classification, will help in producing more generalized set of features and thus reducing the chances of overfitting. Further, due to the usage of the decoder model. the output representation gx(x)) will be forced to learn those features that construct the representative image with no transformations. Representative image of every class has been fixed before the training by analyzing the data corpus. For instance, in the case of MNIST dataset,
Decoder network:

---

Under review as a conference paper at ICLR 2020
Input Image
Input Image
Input-conditioned
Convolution Pooling Layer
Filter generation
Input-aware Filters
Convolution
Deconvolution Layer
Module
Input-can conditioned
Input-conditioned
Feature (a)
Feature
(b)
Figure 2: (a) Proposed convolution module (b) Detailed architecture of input-conditioned filter gen- eration
selected single image corresponding to every digit. This method can be easily extended to include multiple representative images of single class the training process. Further, validate the effectiveness of the decoder network, we selected random set of images from the test set and observed that the constructed images have less transformation when compared with the input image. Figure 3 presents sample input and its corresponding constructed image to clearly depict the reduction in the image with the use of the decoder network. We removed blurriness using some artificial tools to make images more clear.
7
(b)
Figure 3: (a) Input test image (b) Constructed image using the Decoder network
EXPERIMENTS AND DISCUSSION
We analyzed the performance of our proposed network and presented the results of three computer vision datasets. MNIST dataset LeCun & Cortes (2010) variations designed to test rotation and scaling variations have been used for generating experimental results and are compared with state- of-the-art results presented by /Zhang et al. (2018). We also make use of artificially rotated and scaled CIFAR-10 dataset for generating experimental results. To validate the ability to handle different variations, kept our network topology same for different datasets. We have the same number of convolutions and dense layers as compared to Laptev et al. (2016); Zhhang et al. (2018) in order to validate the performance of our proposed convolution framework. For all the experiments, 10% of

---

Under review as a conference paper at ICLR 2020
the training data is separated and used for validation. Also, all the reported results aggregated after multiple runs.
MNIST-Rotation Varriiiaaitiiiiaaitst MNIST LeCun & Cortes (2100) is widely used toy dataset to study modifications proposed in computer vision algorithms. Two MNIST variations: MNIST-rot122 Larochelle al. (2007) and Half-rotated MNIST Jaderberg et al. (2015) exist to analyze the per- formance of networks designed to be invariant to some particular transformations like rotations. As mentioned earlier, we used the same topology check the robustness of our network but with slight change in transformation set so as to remain consistent with prior research.
MNIST-rot- 12k is the most commonly used dataset validate proposed networks with rotation- invariance. It is composed by rotating original MNIST images by a random angle from 0 to 360". contains 12000 instances for training and 50000 for testing. We analyzed the performance of our proposed model on three different transformations set O to compare results with ZZhang et al. (2018). We make use of Dropout layer Srivastavaa et al. (2014) to reduce overfitting and train our network for around 50 epochs, being consistent with prior research.
Table [ presents our experimental results on this dataset where channels denote the size of the transformation set used for single input instance. We achieved consistent improvement when compared with other state-of-the-are approaches. Our proposed model achieves 0.98% error, while best approach in Zhang al. (2018) achieve 1.57% error. It shows that our proposed context-aware convolution filters lead to sigmificant improvement when handling rotation variance present in data.
Half-rotated MNIST is proposed by Jaderberg et al. (2015), mainly due to the small size of MNIST- rot-12k and to address the limitation of images being rotated by more than 90 This dataset contains 60000 training samples and 10000 testing samples which are generated by taking original MNIST dataset and rotating them by random angle from -90 to +900. This makes dataset more close to practical rotation variation scenarios. We achieved 1.12% error with proposed network, where the model is trained for approximately 50 epochs, while best network in Zhang et al. (2018) achieves 1.23% error. The final experimental results and comparison on this dataset are given in Table [
Scaling MNIST: MNIST variations discussed above mostly validates rotation invariance. There- fore, investigate the performance of our proposed approach on other modifications, such as scal- ing, we decided to introduce scaling variation as described in Zhang et al. (2018) artificially. Exper- imental results presented in Table 2 showss a sigmificant improvement in error rate when compared with other -fate-of-the-art results.
CTFAR- The publicly available CIFAR- 10 dataset consists of 60, 60,000 32x32 color images used for object recognition. There are 50,000 training images and 10,000 test images. Each image is assigned to one among ten classes. We artificially introduce rotation following both the strategies employed in generating MNIST-rot- and Half-rotated MNIST. This results in two rotated versions of CIFAR- 10. We make use of TI-POOLING and MINTIN network as our baseline for these experiments. We observe that introduction of proposed convolution framework improves the error rate to 4.3% from 6.4% for rot-1 1k version, and to 5.3% from 7. % for Half-rotated version.
We also experimented with network topologies consisting only multiple layers of our proposed con- volution module in combination with pooling and dense layers, without any standard convolution layer, and achieved similar performance. This indicates that single layer of our proposed convolu- tion module can generate adequate transfornation-inaarinn contextual information present in images, making this module ideal to be used first-layer plugin in any existing CNN network. Similar improvements were observed in the case of scaling CIFAR dataset.
feature representations and incorporate
Filter visualization: To show the effectiveness of our input-conditioned convolution filters, refer Figure 4 we use t-SNE visualization of the input-conditioned filters computed by the filter generation module for images within the test dataset. We randomly selected 200 samples from each class of MNIST-rot-122 test dataset for visualization. It can be easily noticed that filters with the same label make well defined and compact clusters proving our point that for the different input image, distinct filters are generated for better high-level feature extraction.

---

Under review as a conference paper at ICLR 2020
2
-20
-40-
-60
-80
-60
-40
-20
20
60
Figure 4: t-SNE visualization of filter weights
Table I: Error on various rotated data corpus (%)
Corpus
Approach
Channel=4
Channel=7
Channel=8
Channel=13
Channel=24
TI-POOLING
2.47
1.88
1.61
MNIST-rot-12k
MINTIN
1.76
1.59
1.57
ICNN
1.35
1.12
0.98
TI-POOLING
8.7
8.2
7.3
CIFAR-10-rot-I
MINTIN
7.4
6.8
6.4
ICNN
5.3
4.5
4.3
TI-POOLING
1.44
1.46
Half-rotated MNIST
MINTIN
1.32
1.23
ICNN
1.18
1.12
TI-POOLING
8.5
7.9
Half-rotated CIFAR-10
MINTIN
7.2
7.1
ICNN
5.9
5.3
5 CONCLUSION
We introduced a novel input-conditioned convolution filter framework, which unlike traditional CNNs. generates different set of input-aware convolution filters conditioned on input image instance. We also proposed decoder network to mitigate the transformation present the input images with the help of set of pre-defined representative images of the input classes. We employed

---

Under review as a conference paper at ICLR 2020
Table 2: Error various scaled data corpus (%)
Corpus
Approach
Channel=5
Channel=9
TI-POOLING
1.52
1.32
Scaling MNIST
MINTIN
1.01
0.96
ICNN
0.69
0.68
TI-POOLING
7.1
6.8
Scaling CIFAR-I0
MINTIN
6.7
6.3
ICNN
5.6
4.8
our proposed framework in combination with parallel Siamese networks and max max-pooling technique to present cunsformationninvariinn neural network. Our proposed filter generator is, principle, generic representation of standard Convolution layer, which gives our proposed transformation- invariant network greater modeling flexibility. We investigated our proposed approach on the varia- tions of 2 computer vision datasets, which covers both rotation and scaling variations, and achieved state-of-ther of-tthe-rttt results. Our proposed framework can also easily integrated with any CNN model to enhance its ability to incorporate contextual information present in images.
REFERENCES
O. Abdel-Hamid, A. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu. Convolutional neural networks for speech recognition. JEEE/ACM Transactions Audio, Speech, and Language Pro- cessing, 22310))1533-1555 Oct 2014. doi: 10.1099TASSPP.2442233666
Y-Lan Boureau, J. Ponce, and Yann Lecun. recognition. pp. 111-118, 11 2010.
theoretical analysis of feature pooling in visual
Guilhem Chéron, Ivan Laptev, and Cordelia Schmid. P-enn: Pose-based enn features for action recognition. 2015 IEEÉ International Conference Computer Vision (ICCV), pp. 3218 3226, 2015.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei Fei Li. Imagenet: large-scale hierarchical image database. pp. 248 248-255, 06 2009. doi: 10.1 109//C/VP..20999552664
A. Fawzi, H. Samulowitz, D. Turaga, and P. Frossard. Adaptive data augmentation for image clas- sification. In 2016 IEEE International Conference Image Processing (ICIP), pp. 3688-3692, Sep. 2016. doi: 10. 109/IC1122016.5333444
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2015.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. ArXiv, abs/1506.02025, 2015.
D. Laptev and M. Buhmann. Transformation-inaarian convolutional jungles. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 0051-3051, June 2015. doi: 10.1009/CVP.20015.2299
Dmitry Laptev, Nikolay Savinov, Joachim M. Buhmann, and Marc Pollefeys. Ti-pooling: Transformationiinvann pooling for feature learning in convolutional neural networks. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 289-297, 2016.
8

---

Under review as a conference paper at ICLR 2020
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Y. Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. volume 227, pp. 473-480, 01 2007. doi: 10.1 10.1145/1273496.1227 12735555.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 (6(11)22278-2324, Nov 1998. doi: 10. 129157
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. UtLhttp:///an
lecun. wttp///b/////tttt....
Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt, and Gang Hua. A convolutional neural network cascade for face detection. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2015.
David Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60:91-, 11 2004. doi: (1100///::VSSS.00000266666 99615.4,999999949
Karen Simonyan Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1401 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Ma- chine Learning Research, 15:1929-1958 2014. URL http://jmlz org/papers/v15/
Bri arivastaval 1 14ahttll
J. Wu, Yinan Yu, Chang Huang, and Kai Yu. Deep multiple instance learning for image classification and auto-annotation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3460-3469, June 2015. doi: 10.1 1109/CVPR.2015.7298886
Promod Yenigalla, Abhay Kumar, Suraj Tripathi, Chirag Singh, Sibsambhu Kar, and Jithendra Vepa. Speech emotion recognition using spectrogram phoneme embedding. In INTERSPEECH, 2018.
X. Yin and X. Liu. Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions Image Processing, 27(2):964 975, Feb 2018. doi: 10.1 10.1109/11..0017. 2765830.
J. Zhang, K. Jia, P. Yang. F. Qiao, Q. Wei, X. Liu, and H. Yang. Mintin: baxoutt-bsee and input- normalized transformation invariant neural network. 20/8 25th IEEE International Conference on Image Processing (ICIP), pp. 3014-30 3018, Oct 2018, doi: 10.1109//CIP.2011 8.111188144444 1468.


Reviews:
[{"review_id": "rylH3T2dKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Paper summary*\n\nThe authors build an input transformation invariant CNN using the TI-Pooling architecture of (Laptev et al., 2016). They make some modifications to this, namely 1) they replace the convolutional filters, with input-dependent convolutional filters, and 2) they add a decoder network from the final representation, which reconstructs an input transformation rectified image, to encourage the final representation to be fully transformation invariant.\n\n*Paper decision*\n\nI have decided to assign this paper a reject because of two main reasons. \n\n*Supporting arguments*\n\nOne reason is that the base architecture is not novel. This in itself is not a key issue, but I would expect the authors to have done some in depth analysis or experimentation otherwise to compensate for this. I regret, the authors may just not have known that the ideas were already explored in the literature. The second reason is that the work is not well placed in context with prior works. This is both evident in the lack of referenced works (see below for a list) and the lack of sufficient baselines, against which they compare. For instance, if the authors had considered “Learning Steerable Filters for Rotation Equivariant CNNs” by Weiler et al. (2018) they would have known that their MNIST-rot-12k results are not state of the art as they state. In Weiler et al., the authors report 0.714 test set error on MNIST-rot-12k compared to ICNN’s 0.98.\n\nThis all said, I think the paper is well-written and very clear. The structure is straightforward and the experiments seem repeatable from the descriptions made. The stated aims of the paper are also clear: to learn input transformation invariant CNNs using input-conditioned filters.\nUnfortunately a lot of supporting material and prior work has been missed. I list a lot of them here. \n\nWorks on input-conditioned filters and invariance. These are the most important\n\n-Dynamic Steerable Frame Networks, Jacobsen et al., 2017\n-Dynamic Steerable Blocks in Deep Residual Networks., Jacobsen et al., 2017\n\nWorks on input-conditioned filters:\n\n-HyperNetworks, Ha et al., 2016\n-Dynamic Filter Networks, de Brabandere et al., 2016\n\nWorks on invariance:\n\n-Invariance and neural nets, Barnard and Casasent, 1991\n-Group Equivariant Convolutional Networks Cohen and Welling (2015)\n-Harmonic Networks: Deep Translation and Rotation Equivariance: Worrall et al. (2017)\n-Steerable CNNs, Cohen and Welling (2017)\n-Spherical CNNs, Cohen et al. (2018)\n-CubeNet: Equivariance to 3D Rotation and Translation, Worrall and Brostow (2018)\n-Learning steerable filters for rotation equivariant CNNs, Weiler et al. (2018)\n-Gauge Equivariant Convolutional Networks and the Icosahedral CNN, Cohen et al. (2019)\n\n*Questions/notes for the authors*\n\n- Please address the missing references\n- Are the input-conditioned filters conditional on position in the activations, or are they shared across all spatial locations of the image? This is not clear from the text.\n- The image reconstruction reminds me of Transforming Auto-encoders (Hinton et al., 2016) and Interpretable Transformations with Encoder-Decoder Networks (Worrall et al., 2017). How is your setup different?\n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Hygt0c_Jir", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed method in this paper tries to make the CNN robust to the input image transformation by learning to generate convolutional filters. \nThe proposed architecture has two main parts. \n1) Filter Generation:  Given an input image, a set of predefined transformations are applied to the image. After extracting features from the input transformed images with the Siames network, a set of Convolutional filter are estimated. The idea is that these input-dependent filters can compensate all of  transformation in the image.\n2) Classification and reconstruction part: The generated convolutional filters are applied to the image and after extracting deeper features a representation vector is computed. This representation vector will be used for classification and reconstruction of the input image to make sure that it has all of the necessary information. \n\nPositive points:\n1) The writing is clear.\n\nNegative points:\nThe proposed method is not novel. The proposed method will be robust to the transformations that are used during training but it cannot generalize to other useen transformations.\n2) The experimental results are weak, the authors should compare their method on more difficult datasets like ImageNet dataset. \n3) The authors should compare their proposed method with the \"Spatial transformer networks, NIPS 2015.\" in detail. \n\nIn conclusion, my recommendation for this paper is \"weak reject\".\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g0BlpjKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an Input-conditioned Convolutional Neural Network (ICNN) to automatically impose transformation-invariance. The contribution of the manuscript is two-folds\n(a) After transforming the input using a pre-determined set of transformations, a set of input-conditioned filter generators are used (and trained) to cater to different input contents.\n(b) A decoder is used after the max-pooling layer (of the  Siamese network.) And an L-2 reconstruction loss (with respect to a chosen class representative) is added to the cross-entropy loss for classification.\n\nOverall the paper is well written, and it is fairly easy to read. However, I am not totally convinced that the two contributions of the paper are significant to transformation-invariant representations, and my reasonings are follows\n\n1. Why is a decoder needed in the architecture? If the objective is to achieve transformation-invariance, one can easily compare the L-2 distance between the max-pooled feature maps of a given input to that of the class representative. Why bother using a decoding architecture?\n2. Choosing a \"class representative\" in the CNN seems very restrictive. Why if the underlying task is not image classification? Besides, I am very curious about the experiment on the CIFAR-10 dataset: do the constructed images of all test samples look like the one chosen class representative in the training data? (i.e., compared to figure 3)\n3. The input-conditioned filter generation seems a little confusing. Is this what you want to achieve? Say if the pre-determined transformations are rotation (scaling), then the input-conditioned filter should be generated as rotated (scaled) version of the same filters? If so, why not just rotate (rescale) the filters? There are lots of group-equivariant CNNs that have been proposed before for such effect. Besides, I am confused why fractionally-strided convolutions are used for filter generation?\n\nOther comments:\n1. The reference for fractionally-strided convolutions should be fixed.\n2. Why there is no bias term in convolutional modules (page 4, second paragraph?)\n3. What does ICNN short for? The first appearance of the abbreviation in the abstract needs more explanation.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a CNN that is invariant to input transformation, by making two modifications on top of the TI-pooling architecture: the input-dependent convolutional filters, and a decoder network to ensure fully transformation invariant. Reviewer #1 concerns the limited novelty, unconvincing experimental results. Reviewer #2 praises the paper being well written, but is not convinced by the significance of the contributions. The authors respond to Reviewer #2 but did not change the rating. Reviewer #3 especially concerns that the paper is not well positioned with respect to the related prior work.  Given these concerns and overall negative rating (two weak reject and one reject), the AC recommends reject.
Decision: Reject