# AWEncoder: Adversarial Watermarking Pre-Trained Encoders in Contrastive Learning
## 1 Introduction  
The rapid development of deep learning (DL) has benefited a lot from a large number of diverse labeled datasets, which, however, simultaneously has become a factor hindering the further development of DL. It is due to the reason that the cost of collecting sufficient high-quality datasets with correct labels is prohibitively expensive. Fortunately, the emergence of self-supervised learning (SSL) [1] to train a powerful encoder with the unlabeled samples can effectively overcome the above obstacle. Contrastive learning [2; 3], as one mainstream self-supervised learning techniques, has achieved great success in various tasks. For example, SimCLR [4] and MoCo [5] have already enabled the SSL encoders to outperform traditional supervised learning-based encoders in several downstream tasks. However, training the encoders still requires lots of unlabeled data and consumes a great deal of computing resources. Many malicious users are likely to steal the pre-trained encoders to obtain illegal income and even bring serious security risks [6; 7]. Therefore, we need to find reliable solutions to protect the intellectual property of the pre-trained encoders. As a major technique to protect the intellectual property of DL models, model watermarking [8; 9; 10; 11] has recently been widely studied. It can be roughly divided into two categories: _white-box watermarking_ and _black-box watermarking_. The former [12; 13] embeds a watermark into the internal parameters [14], feature maps [15] or structures [16] of a DLmodel, which needs white-box access to the watermarked model. The latter [17] usually uses backdooring [18; 19] or similar techniques, such as adversarial attack [20], to mark a model. The ownership can be verified by analyzing the classification results of the marked model in correspondence to a set of carefully crafted samples. Compared with white-box watermarking, black-box watermarking is more desirable for applications, since in most cases, the model owner has no access to the internal details of the target model. However, for protecting the aforementioned encoders, the lack of a priori knowledge of downstream tasks makes it quite difficult to craft the special samples for black-box watermarking. It indicates that we cannot simply extend the existing black-box watermarking schemes to the encoders. As a result, we urgently need to develop novel watermarking schemes specifically for the pre-trained encoders. In this paper, we introduce AWEncoder, a copyright protection method for the pre-trained encoders in contrastive learning via an adversarial watermark. In the proposed method, through optimizing an adversarial perturbation [21], we determine such a perturbation according to the pre-trained encoder that it can cluster the perturbed training samples around a randomly selected key image in the embedding space and can also be used as the secret watermark. The watermark is then embedded into the pre-trained encoder via contrastive learning by further optimizing a joint loss involving watermark embedding. In this way, even if the downstream task of the watermarked encoder is unknown to us, we can still verify its ownership under the black-box condition. Experimental results demonstrate that the ownership can be reliably verified under both white-box and black-box conditions, which is quite helpful for applications. In summary, the main contributions of this paper include:  
* We propose a novel adversarial watermarking strategy for the pre-trained encoders in the embedding space, which is more effective for watermark verification compared with previous black-box watermarking methods. * Unlike conventional watermarking methods assuming that the downstream task of the watermarked model is the same as the original one, the proposed method does not require a priori knowledge of the downstream task. As a result, the proposed method enables us to verify the ownership under white-box and stricter black-box settings, which is more applicable to practice compared with previous ones.