###### Abstract
To find better constellation schemes, an improved pairwise optimization (PO) algorithm was proposed to design a multi-dimensional constellation, which is based on minimizing the analytical BER of given M constellation points and bit mapping [20]. However, the constellation design generated by the PO algorithm is always based on prior theoretical knowledge, which cannot guarantee optimal performance in practical systems due to nonlinearity or other impairments. In other words, there is always a gap between the performance of current shaping schemes based on theoretical models and the optimal performance of real systems. Thus, designing a constellation-shaping scheme that is adaptive and closer to the specific real system is a critical issue in LED-based VLC systems. With the development of deep learning, the concept of end-to-end communication was proposed to coordinate the optimization between transmitter and receiver in the traditional communication model, which is considered as a communication method that achieves global optima [21]. Based on end-to-end communication, novel autoencoder-based learning of joint geometric and probabilistic constellation shaping for coded modulation systems was proposed [22]. Geometrically shaped multi-dimensional modulation formats are designed via deep learning methods [23], and an end-to-end-based multi-dimensional GS strategy is proposed to achieve phase noise robustness [24]. Additionally, a differentiable GS method based on a blind phase search was also proposed for phase noise channels [25]. For visible-light communication, end-to-end communication based on an autoencoder has also gradually become a hot research topic. An autoencoder based on a data-driven channel model used to optimize end-to-end VLC systems has been proposed [10; 26; 27; 28; 29; 30]. In [30], a deep-learning-based E2E VLCnet was suggested to realize flicker reduction and dimming operation. Since most previous studies have only been validated with a simulation experiment, in [29], an autoencoder was deployed in a real VLC system for the first time and successfully achieved a transmission rate of 1.875 Gbps under the 7% HD-FEC threshold. However, this work cannot further improve the transmission rate, because it discards almost all communication techniques and digital signal processing methods, only using two neural networks to send and receive data. How to combine autoencoder and digital signal processing algorithms to achieve jointly optimal communication performance is still a great challenge. In this work, for the first time, we propose and experimentally demonstrate an autoencoder-based geometric shaping model (AEGSM) framework to jointly optimize quadrature amplitude modulation (QAM) signals at the symbol-wise and bit-wise levels for underwater visible-light communication. Unlike traditional GS methods, which only give theoretically optimal shaping solutions, our framework can always obtain the globally optimal shaping scheme for a specific channel or different application scenarios. Compared with the conventional autoencoder-based geometric shaping model or other geometric shaping methods, our framework realizes the effective combination of end-to-end learning and traditional geometric shaping theory so that the obtained constellation scheme can further guarantee optimal BER performance on the basis of ensuring optimal SER performance. And due to its adaptability, it is suitable for any channel condition or application scenario, bringing new inspiration to the other communication fields. In a real UVLC system, a 2.05 Gbps transmission is achieved under the hard decision-forward error correction (HD-FEC) threshold of \(3.8\times 10^{-3}\) by employing the autoencoder-based 8 QAM (AE-8 QAM) optimized by the AEGSM, which is 103 Mbps faster than the Norm-8 QAM. The AE-8 QAM also shows its resistance to nonlinearity and enables the UVLC system to operate within a larger dynamic range of driving voltages, extending it by 58.37%.