Robust feature space separation for deep convolutional neural network training

Abstract This paper introduces two deep convolutional neural network training techniques that lead to more robust feature subspace separation in comparison to traditional training. Assume that dataset has M labels. The first method creates M deep convolutional neural networks called $$\{\text {DCNN}_i\}_{i=1}^{M}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msubsup><mml:mrow><mml:mo>{</mml:mo><mml:msub><mml:mtext>DCNN</mml:mtext><mml:mi>i</mml:mi></mml:msub><mml:mo>}</mml:mo></mml:mrow><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>M</mml:mi></mml:msubsup></mml:math> . Each of the networks $$\text {DCNN}_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mtext>DCNN</mml:mtext><mml:mi>i</mml:mi></mml:msub></mml:math> is composed of a convolutional neural network ( $$\text {CNN}_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mtext>CNN</mml:mtext><mml:mi>i</mml:mi></mml:msub></mml:math> ) and a fully connected neural network ( $$\text {FCNN}_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mtext>FCNN</mml:mtext><mml:mi>i</mml:mi></mml:msub></mml:math> ). In training, a set of projection matrices $$\{\mathbf {P}_i\}_{i=1}^M$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msubsup><mml:mrow><mml:mo>{</mml:mo><mml:msub><mml:mi>P</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mo>}</mml:mo></mml:mrow><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>M</mml:mi></mml:msubsup></mml:math> are created and adaptively updated as representations for feature subspaces $$\{\mathcal {S}_i\}_{i=1}^M$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msubsup><mml:mrow><mml:mo>{</mml:mo><mml:msub><mml:mi>S</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mo>}</mml:mo></mml:mrow><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>M</mml:mi></mml:msubsup></mml:math> . A rejection value is computed for each training based on its projections on feature subspaces. Each $$\text {FCNN}_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mtext>FCNN</mml:mtext><mml:mi>i</mml:mi></mml:msub></mml:math> acts as a binary classifier with a cost function whose main parameter is rejection values. A threshold value $$t_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi>t</mml:mi><mml:mi>i</mml:mi></mml:msub></mml:math> is determined for $$i^{th}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msup><mml:mi>i</mml:mi><mml:mrow><mml:mi>th</mml:mi></mml:mrow></mml:msup></mml:math> network $$\text {DCNN}_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mtext>DCNN</mml:mtext><mml:mi>i</mml:mi></mml:msub></mml:math> . A testing strategy utilizing $$\{t_i\}_{i=1}^M$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msubsup><mml:mrow><mml:mo>{</mml:mo><mml:msub><mml:mi>t</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mo>}</mml:mo></mml:mrow><mml:mrow><mml:mi>i</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow><mml:mi>M</mml:mi></mml:msubsup></mml:math> is also introduced. The second method creates a single DCNN and it computes a cost function whose parameters depend on subspace separations using the geodesic distance on the Grasmannian manifold of subspaces $$\mathcal {S}_i$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mi>S</mml:mi><mml:mi>i</mml:mi></mml:msub></mml:math> and the sum of all remaining subspaces $$\{\mathcal {S}_j\}_{j=1,j\ne i}^M$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msubsup><mml:mrow><mml:mo>{</mml:mo><mml:msub><mml:mi>S</mml:mi><mml:mi>j</mml:mi></mml:msub><mml:mo>}</mml:mo></mml:mrow><mml:mrow><mml:mi>j</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn><mml:mo>,</mml:mo><mml:mi>j</mml:mi><mml:mo>≠</mml:mo><mml:mi>i</mml:mi></mml:mrow><mml:mi>M</mml:mi></mml:msubsup></mml:math> . The proposed methods are tested using multiple network topologies. It is shown that while the first method works better for smaller networks, the second method performs better for complex architectures.