## 1 Introduction  
Coherent optical communications have been proposed as a viable solution for maximising the signal capacity in both short-reach and long-haul communications [1]. However, Kerr-induced fiber nonlinearity prevents channel capacity from approaching the Shannon limit, especially when the signal power is high [2]. Endeavours to surpass the Kerr nonlinearity limit have been performed by techniques that in principle compensate deterministic nonlinearities [3]. For example, nonlinearities can be combated by either inserting an optical phase conjugator (OPC) at the middle point of the link [4], or by inverting the fiber effects among multiple frequency stabilised optical signals [5]. However, OPC reduces the flexibility in an optically routed network, whereas in [5], a digital back propagation (DBP) [6] pre-compensator was used, which is of excessive complexity. Other famous techniques include hybrid pre- and post-compensation [7], Volterra-based nonlinear equalisation (NLE) [8], phase-conjugated twin waves (PC-TW) [9], and the nonlinear Fourier transform (NFT) [10]. Unfortunately, pre-/post-compensation algorithms and Volterra-NLE present marginal performance enhancement, PC-TW sacrifices signal capacity and NFT is unpractical for real-time signal processing. Above all, the aforementioned deterministic methods are unable to tackle stochastic nonlinearities, such as the amplified spontaneous emission (ASE) noise induced from optical amplifiers.  
Unsupervised machine learning clustering has been recently introduced in optical communications for blind (training data-free) nonlinear equalisation (BNLE). Such unsupervised algorithms can tackle stochastic nonlinearities and include, for example, fuzzy logic C-means [11], K-means [11; 12], hierarchical [11], affinity propagation [13], and Gaussian mixture [14] clustering. The algorithms used in [11; 12; 13; 14] are mainly used for compensating fiber nonlinearity in long-haul transmission systems, in which the distorted received constellation diagrams contain Gaussian-circular clusters. However, there is an uncertainty if the machine learning clustering algorithms can be effective for non-circular rotated clusters caused by the strong nonlinear phase noise.  
In this work, the aforementioned issue is addressed by experimentally demonstrating the first BNLE that harnesses the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [15] algorithm in 40 Gb/s 16 quadrature amplitude modulation (QAM) coherent optical signals being transmitted at 50 km. As a proof of concept, DBSCAN is tested for very high launched optical powers (LOPs), where the clusters of the received constellation diagrams are vastly rotated by means of self-phase modulation (SPM). Two modified DBSCAN methods are also adopted, in which the \"un-clustered\" noisy points are further processed using (1) K-means and (2) the minimum distance between an unlabelled point and the clustered points. It is shown that DBSCAN offers up to 0.83 dB Q-factor improvement over K-means and 8.84 dB when compared to linear equalisation at +16 dBm of LOP. This occurs because DBSCAN can effectively recover non-circularly-symmetric (elliptical form) noisy clusters by effectively combating SPM.