###### Abstract  
M-ary pulse-amplitude modulation (PAM) meets the requirements of data center communication because of its simplicity, but coarse entropy granularity cannot meet the dynamic bandwidth demands, and there is a large capacity gap between uniform formats and the Shannon limit. The dense wavelength division multiplexing (DWDM) system is widely used to increase the channel capacity, but low spectral efficiency of the intensity modulation/direct detection (IM/DD) solution restricts the throughput of the modern DWDM data center networks. Probabilistic shaping distribution is a good candidate to offer us a fine entropy granularity and efficiently reduce the gap to the Shannon limit, and Nyquist pulse shaping is widely used to increase the spectral efficiency. We aim toward the joint usage of probabilistic shaping and Nyquist pulse shaping with low-density parity-check (LDPC) coding to improve the bit error rate (BER) performance of 8-PAM signal transmission. We optimized the code rate of the LDPC code and compared different Nyquist pulse shaping parameters using simulations and experiments. We achieved a 0.43 dB gain using Nyquist pulse shaping, and a 1.1 dB gain using probabilistic shaping, while the joint use of probabilistic shaping and Nyquist pulse shaping achieved a 1.27 dB gain, which offers an excellent improvement without upgrading the transceivers. p 1 2019  
schemes can approach the Shannon limit. Since it may be easier to have a common standard agreement among network service providers if the PS scheme is used for constellation shaping, PS should be more suitable to be used in data center networks. PS can be realized using Huffman coding [28; 29], many-to-one mapping [30], and a constant composition distribution matcher (CCDM) [31; 32]. Given that a CCDM can flexibly generate fractional entropy without systematic error, the optical industry is more inclined to apply a CCDM-based PS scheme. PS can not only be used for quadrature amplitude modulation (QAM) formats, but also for a PAM scheme [33; 34; 35]. For a PAM scheme, it imposes an exponential-like distribution on a set of equidistant constellation points. It transmits symbols with smaller amplitudes more often than larger ones, which can offer us a fine entropy granularity and enable a transmission with a lower signal-to-noise ratio (SNR) at the same forward error correction (FEC) overhead. The requirement of channel capacity is greatly increasing nowadays, which leads to the dense wavelength division multiplexing (DWDM) system being widely using in data center networks [36; 37; 38], but the low spectral efficiency of the intensity modulation/direct detection (IM/DD) solution restricts the throughput of the modern DWDM data center networks, which results in large investments to upgrade the transceivers in order to meet the increasing bandwidth requirement. Nyquist pulse shaping (NPS) is a good solution to increase the spectral efficiency of a DWDM system [39; 40]; it uses a raised-cosine filter to limit the effective bandwidth and can reduce the inter-symbol interference (ISI) by properly selecting the roll-off factor (ROF) for NPS. The purpose of this paper was to demonstrate the joint usage of both the PS distribution and NPS in a DWDM system for short-reach applications, in particular data center networks. In addition to the joint shaping format, the employment of a suitable FEC code was also important for improving the overall performance. Low-density parity-check (LDPC) codes represent excellent FEC candidates to be applied together with the proposed shaping scheme. We transmitted LDPC-coded 8-PAM signals for both uniform and nonuniform signaling and compared the performance for different LDPC code rates. We further evaluated the performance improvements when NPS was used for different ROFs in both PS and uniform distribution-based systems. We experimentally evaluated the bit error rate (BER) performance improvement of the proposed joint shaping scheme, compared with the uniform signaling scheme. The rest of the paper is organized as follows.