## 2 Proposed Framework
### NN Design via Hankelization-Based Preprocessing
### Empirical Latent Space Comparison Across Models

To investigate the feature representation capability of each model, we empirically visualize the latent vectors \(\mathbf{z}\in\mathbb{R}^{3}\) obtained from the final hidden layer prior to classification. Let \(\mathcal{G}\) denote the set of considered models, including the proposed methods and three baseline schemes. For each model \(g\in\mathcal{G}\), the optimized NN model is denoted by \(\mathcal{N}_{g}^{*}\), which can be decomposed as

\[\mathcal{N}_{g}^{*}=\mathcal{C}_{g}\circ f_{\text{latent}}^{(g)} \tag{13}\]

where \(\circ\) is the composition operator of the function, \(f_{\text{latent}}^{(g)}(\cdot)\) denotes the feature mapping function up to the penultimate layer, and \(\mathcal{C}_{g}(\cdot)\) is the final classification layer of model \(g\). Given a model-specific input \(\mathbf{y}\), the corresponding latent feature vector is obtained as

\[\mathbf{z}=f_{\text{latent}}^{(g)}(\mathbf{y}). \tag{14}\]

To facilitate visualization, the output dimension of \(f_{\text{latent}}^{(g)}\) was fixed to 3 for all \(g\in\mathcal{G}\) in this analysis. Note, however, that the original models used for classification in the next subsection employ higher-dimensional latent spaces. Figure 4 shows the resulting 3D scatters of latent vectors \(\mathbf{z}\in\mathbb{R}^{3}\), categorized by modulation classes, i.e., BPSK, QPSK, 16 QAM, and 64 QAM. Each subfigure corresponds to a different model \(g\in\mathcal{G}\). As seen in Figure 3(a),b, the proposed methods result in more structured and discriminative embeddings for modulation types, whereas the baseline models exhibit more entangled and less separable latent representations. ### Performance Evaluation for Modulation Recognition

In Figure 5, confusion matrices are shown for six methods, each evaluated at 10 dB SNR conditions across four modulation schemes. In each matrix, rows indicate the ground-truth modulation classes, whereas columns correspond to the predicted classes. For instance, as seen in Figure 5a a value of 94.0% in the third row and third column indicates that 94.0% of 16 QAM samples were correctly classified as 16 QAM, whereas 6.0% were incorrectly classified as 16 QAM, and so forth. As shown in Figure 5b, the proposed method (with the FFT-processed signal) achieves the highest classification accuracy across almost all modulation types compared to other baseline methods. Among the baseline methods, SCNN2 in Figure 5d shows competitive performance, likely due to its spectrogram-based preprocessing with Gaussian filtering, which effectively suppresses noise. While the proposed method with original signals (Figure 5a) does not employ explicit denoising, it still achieves high accuracy for BPSK, 16 QAM, and 64 QAM, while exhibiting limited discriminability between QPSK and BPSK. Figure 6 shows the detection rate according to the number of training datasets. For instance, a total of 2800 training samples corresponds to 700 samples per modulation class when four modulation types are considered. Across all SNR levels and training sizes, the proposed method (with FFT-processed signal), shown in orange, consistently achieves the highest detection rate. As shown in Figure 6a, even when trained with only 560 samples in total--corresponding to 140 samples per class under the SNR condition of 4 dB--the proposed method (with FFT-processed signal) maintains a detection rate approaching 0.9, demonstrating strong robustness under limited data availability. Figure 4: Data visualization based on latent space (SNR = 10 dB). (**a**) Proposed method (with original signal). (**b**) Proposed method (with FFT-processed signal). (**c**) Deep learning (CNN with real and imaginary values of signal). (**d**) SCNN2. (**e**) CLDNN. (**f**) LSTMDAE. Table 3 presents the average detection rate and corresponding standard deviation across 10 independent trials, each conducted with different random seeds, for various SNR levels ranging from \(-20\) to 18 dB.