SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jan Dubinski  $^{1,2*}$  Stanisław Pawlak  $^{1\dagger}$  Franziska Boenisch  $^{4\dagger}$  Tomasz Trzeciński  $^{1,2,3}$  Adam Dziedzic  $^{4*‡}$ $^{1}$ Warsaw University of Technology  $^{2}$ IDEAS NCBR  $^{3}$ Tooploox  $^{4}$ CISPA Helmholtz Center for Information Security
Machine Learning as a Service (MLaaS) APIs provide ready-to-use and high-utility encoders that generate vector representations for given inputs. Since these encoders are very costly to train, they become lucrative targets for model stealing attacks during which an adversary leverages query access to the API to replicate the encoder locally at a fraction of the original training costs. We propose *Bucks* for *Buckets* (*B4B*), the first active defense that prevents stealing while the attack is happening without degrading representation quality for legitimate API users. Our defense relies on the observation that the representations returned to adversaries who try to steal the encoder's functionality cover a significantly larger fraction of the embedding space than representations of legitimate users who utilize the encoder to solve a particular downstream task. B4B leverages this to adaptively adjust the utility of the returned representations according to a user's coverage of the embedding space. To prevent adaptive adversaries from eluding our defense by simply creating multiple user accounts (sybils), B4B also individually transforms each user's representations. This prevents the adversary from directly aggregating representations over multiple accounts to create their stolen encoder copy. Our active defense opens a new path towards securely sharing and democratizing encoders over public APIs.<sup>5</sup>
In model stealing attacks, adversaries extract a machine learning model exposed via a public API by repeatedly querying it and updating their own stolen copy based on the obtained responses. Model stealing was shown to be one of the main threats to the security of machine learning models in practice [38]. Also in research, since the introduction of the first extraction attack against classifiers [40], a lot of work on improving stealing [27, 33, 40, 41], extending it to different model types [8, 37], and proposing adequate defenses [18, 25, 26, 31] has been put forward. With the recent shift in learning paradigms from supervised to self supervised learning (SSL), especially the need for new defenses becomes increasingly pressing. From an academic viewpoint, the urge arises because it was shown that SSL models (encoders) are even more vulnerable to model stealing [16, 29, 36] than their supervised counterparts. This is because whereas supervised models' output is low dimensional, e.g., per-class probabilities or pure labels, SSL encoders output high-dimensional representation vectors that encode a larger amount of information and thereby facilitate stealing. In addition, from a practical industry's viewpoint, defenses are required since many popular API providers, such as Cohore, OpenAI, or Clarify [1-3] already expose their high-value SSL encoders via APIs to a broad range of users.
*Corresponding authors: jan.dubinski.dokit@pw.edu.pl and adam.dziedzic@cispa.de
Equal contribution.
$\ddagger$ Project Lead.
$^{5}$ Code available at https://github.com/stapaw/b4b-active-encoder-defense
Figure 1: Overview of B4B. In the upper part, we present our B4B framework that consists of three modular building blocks: (1) A coverage estimation to track the fraction of embedding space covered by the representations returned to each user, (2) a cost function that serves to map the coverage to a concrete penalty to prevent stealing, and (3) per-user transformations that are applied to the returned representations to prevent sybil attacks. In the lower part, we present a concrete instantiation of B4B and the operation flow of our defense: ① The API calculates representations for the incoming queries. ② We instantiate the coverage estimation with local sensitive hashing and estimate the covered space as the fraction of hash buckets occupied. We calibrate the costs by adding noise to the representations according to the coverage. ③ We apply a set of transformations on a per-user basis. ④ The noised and transformed representations are returned to the user.
Most of the current defenses against encoder stealing are reactive, i.e., they do not actively prevent the stealing but rather aim at detecting it by adding watermarks to the encoder [14, 16] or performing dataset inference to identify stolen copies [17]. Since at the point of detection, the damage of stealing has already been inflicted, we argue that reactive defenses intervene too late and we advocate for active defenses that prevent stealing while it is happening. Yet, active defenses are challenging to implement because they not only need to prevent stealing but also should preserve the utility of representations for legitimate users. The only existing active defense against encoder stealing [29] falls short on this latter aspect since it significantly degrades the quality of representations for all users.
To close the gap between required and existing defenses, we propose *Bucks for Buckets* (B4B), the first active defense against encoder stealing that does not harm utility for legitimate users. B4B leverages the observation that the representations returned to adversaries who try to steal the encoder's functionality cover a significantly larger fraction of the full embedding space than representations of legitimate users who utilize the encoder to solve a particular downstream task. To turn this observation into a practical defense, B4B is equipped with three modular building blocks: (1) The first building block is a tracking mechanism that continuously estimates the fraction of the embedding space covered by the representations returned to each user. The intuition why this is relevant is that by covering large fractions of the embedding space, the representations will suffice for an adversary to reproduce the encoder's functionality, i.e., to successfully steal it. (2) B4B's second building block consists of a cost function to translate the covered fraction of the embedding space into a concrete penalty. We require this cost function to significantly penalize adversaries trying to steal the model while having only a minimal effect on legitimate users. (3) The third building block contains transformations that can be applied to the representations on a per-user basis to prevent adaptive attackers from circumventing our defense by creating multiple user accounts (sybils) and distributing their queries over these accounts such that they minimize the overall cost. We present the different building blocks of B4B in Figure 1.
While B4B's modularity enables different instantiations of the three building blocks, we propose a concrete end-to-end instantiation to showcase the practicability of our approach. To implement tracking of the covered embedding space, we employ local sensitive hashing that maps any representation returned to a given user into a set of hash buckets. We base our cost function (i.e., the "bucks") on utility and make B4B add noise to the representations with a magnitude that increases with the number of buckets occupied by the given user. While the scale of noise added to legitimate users'
representations does not harm their downstream performance due to their small embedding space coverage, the representations returned to an adversary become increasingly noisy—significantly degrading the performance of their stolen encoder. Finally, we rely on a set of transformations (e.g., affine transformations, shuffling, padding) that preserve downstream utility [17]. While, as a consequence, legitimate users remain unaffected by these transformations, adversaries cannot directly combine the representations obtained through different sybil accounts anymore to train their stolen copy of the encoder. Instead, they first have to remap all representations into the same embedding space, which we show causes both query and computation overhead and still reduces the performance of the stolen encoder.
1. We present B4B, the first active defense against encoder stealing that does not harm legitimate users' downstream performance. B4B's three building blocks enable penalizing adversaries whose returned representations cover large fractions of the embedding space and prevent sybil attacks.
2. We propose a concrete instantiation of B4B that relies on local sensitive hashing and decreases the quality of representations returned to a user once their representations fill too many hash buckets.
3. We provide an end-to-end evaluation of our defense to highlight its effectiveness in offering high utility representations for legitimate users and degrading the performance of stolen encoders in both the single and the sybil-accounts setup.
Model Extraction Attacks. The goal of the model extraction attacks is to replicate the functionality of a victim model  $f_{v}$  trained on a dataset  $D_{v}$ . An attacker has a black box access to the victim model and uses a stealing dataset  $D_{s} = \{q_{i},f_{v}(q_{i})\}_{i = 1}^{n}$ , consisting of queries  $q_{i}$  and the corresponding outputs  $f_{v}(q_{i})$  returned by the victim model, to train a stolen model  $f_{s}$ . Model extraction attacks have been shown against various types of models including classifiers [24, 40] and encoders [16, 36].
Self Supervised Learning and Encoders. SSL is an increasingly popular machine learning paradigm. It trains encoder models to generate representations from complex inputs without relying on explicit labels. These representations encode useful features of a given input, enabling efficient learning for multiple downstream tasks. Many SSL frameworks have been proposed [9, 10, 12, 22, 23, 44]. In our work, we focus on the two popular SSL vision encoders, namely SimSiam [12] and DINO [9], which return high-quality representations that achieve state-of-the-art performance on downstream tasks when assessed by training a linear classifier directly on representations. SimSiam trains with two Siamese encoders with directly shared weights. A prediction MLP head is applied to one of the encoders  $f_{1}$ , and the other encoder  $f_{2}$  has a stop-gradient, where both operations are used for avoiding collapsing solutions. In contrast, DINO shares only architecture (not weights) between a student  $f_{1}$  and a teacher model  $f_{2}$ , also with the stop-gradient operation, but not the prediction head. While SimSiam uses convolutional neural networks (CNNs), DINO also employs vision transformers (ViTs). Both frameworks use a symmetrized loss of the form  $\frac{1}{2} g(f_1(x_1),f_2(x_2)) + \frac{1}{2} g(f_1(x_2),f_2(x_1))$  in their optimization objectives, where  $g(\cdot ,\cdot)$  is negative cosine similarity for SimSiam and cross-entropy for DINO. SimSiam and DINO's similarities and differences demonstrate our method's broad applicability across SSL frameworks. More details can be found in Appendix E.
Stealing Encoders. The stealing of SSL encoders was shown to be extremely effective [16, 29, 36]. The goal of extracting encoders is to maximize the similarity of the outputs from the stolen local copy and the original representations output by the victim encoder. Therefore, while training the stolen copy, the adversary either imitates a self-supervised training using a contrastive loss function, e.g., InfoNCE [10] or SoftNN [21] or directly matches both models' representations via the Mean Squared Error (MSE) loss. To reduce the number of queries sent to the victim encoder, the attack proposed in [29] leverages the key observation that the victim encoder returns similar representations for any image and its augmented versions. Therefore, a given image can be sent to the victim while the stolen copy is trained using many augmentations of this image, where the representation of a given augmented image is approximated as the one of the original image produced by the victim encoder.
Defending Encoders. Recently, watermarking [7, 25, 42] methods have been proposed to detect stolen encoders [14, 16, 43]. Many of these approaches use downstream tasks to check if a watermark embedded into a victim encoder is present in a suspect encoder. Dataset inference [30] is another type
of encoder ownership resolution. It uses the victim's training dataset as a unique signature, leveraging the following observation: for a victim encoder trained on its private data as well as for its stolen copies, the distribution of the representations generated from the victim's training data differs from the distribution of the representations generated on the test data. In contrast, for an independently trained encoder, these two distributions cannot be distinguished, allowing the detection of stolen copies [17]. However, all the previous methods are reactive and aim at detecting the stolen encoder instead of actively preventing the attack. The only preliminary active defenses for encoders were proposed by [16, 29]. They either perturb or truncate the answers to poison the training objective of an attacker. These operations were shown to harm substantially the performance of legitimate users, which renders the defense impractical. In contrast, our B4B has negligible impact on the quality of representations returned to legitimate users.
3 Actively Defending against Model Stealing with B4B
B4B aims at actively preventing model stealing while preserving high-utility representations for legitimate users. Before introducing the three main building blocks of B4B, namely (1) the estimation of embedding space coverage, (2) the cost function, and (3) the transformation of representations (see Figure 1), we detail our threat model and the observation on embedding space coverage that represents the intuition behind our approach.
3.1 Threat Model and Intuition
Our setup and the resulting threat model are inspired by public APIs, such as Cohere, OpenAI, or Clarify [1-3] that expose encoders to users through a pre-defined interface. These encoders are trained using SSL on large amounts of unlabeled data, often crawled from the internet, and therefore from diverse distributions. We notice that to provide rich representations to multiple users, the training dataset of the encoder needs to be significantly more diverse than the individual downstream tasks that the users query for representations. For instance, if the encoder behind the API is trained on the ImageNet dataset, then the legitimate users are expected to query the API for downstream tasks, such as CIFAR10 or SVHN. Similarly, if the encoder is trained on CIFAR10, the expected downstream tasks are MNIST or Fashion MNIST. Yet, in the design of our defense, we consider adversaries who can query the encoder with arbitrary inputs to obtain high-dimensional representation vectors from the encoder. Our defense is independent of the protected encoder's architecture and does not rely on any assumption about the adversary's data and query strategy.
We argue that even in this restricted setup, our defense can distinguish between adversaries and legitimate users by analyzing the distribution of representations returned to them. In Figure 2, by using PCA to project representations for different datasets to a two-dimensional space, we visualize that representations for different downstream tasks cluster in disjoint and small sub-spaces of the full embedding space. The representations were obtained from a SimSiam encoder originally trained on ImageNet (we observe similar clustering for DINO shown in Appendix F). As a result, legitimate users can be characterized by their representations' small coverage of the embedding space. In contrast, the adversary does not aim at solving a particular downstream task. They instead would want to obtain representations that cover large fractions of the embedding space. This enables reproducing the overall functionality of the encoder (instead of only learning some local task-specific behavior). Indeed, it has been empirically shown by prior work, such as [16], that stealing with multiple
distributions, e.g., by relying on the complex ImageNet dataset, yields higher performance of the stolen encoder on various downstream tasks than stealing with a downstream dataset, such as CIFAR10. As a result, intuitively, we can identify and penalize adversaries based on their coverage of the embedding space, which will be significantly larger than the coverage of legitimate users. We leverage this intuition to build our B4B defense and present our three main building blocks in the following sections.
Figure 2: Representations from Different Tasks Occupy Different Sub-Spaces of the Embedding Space. Presented for Fashion-MNIST, SVHN, CIFAR10, and STL10.
3.2 Building Block 1: Coverage Estimation of the Embedding Space
The first building block of our B4B serves to estimate and continuously keep track of the fraction of the embedding space occupied by any given user. Let  $\mathcal{E}$  denote our embedding space of dimension  $s$ , further, let  $U$  be a user with a query dataset  $D = q_{1},\ldots ,q_{n}\in \mathcal{D}$  and let  $f_{v}:\mathcal{D}\to \mathcal{E}$  be our protected victim encoder that maps data points from the input to the embedding space. Assume user  $U$  has, so far, queried a subset of their data points  $q_{1},\dots,q_{j}$  with  $j\leq n$  to the encoder and obtained the representations  $r_1,\ldots ,r_j$  with each  $r_i\in \mathbb{R}^s$ . We aim to estimate the true fraction of the embedding space  $\mathcal{E}_f^U$  that is covered by all returned representations  $r_1,\ldots ,r_j$  to user  $U$  and denote our estimate by  $\tilde{\mathcal{E}}_f^U$ .
Local Sensitive Hashing. One of the methods to approximate the occupied space by representations returned to a given user is via Local Sensitive Hashing (LSH) [39]. We rely on this approach for the concrete instantiation of our B4B and use it to track per-user coverage of the embedding space. Standard (cryptographic) hash functions are characterized by high dispersion such that hash collisions are minimized. In contrast, LSH hashes similar data points into the same or proximal, so-called hash buckets. This functionality is desired when dealing with searches in high-dimensional spaces or with a large number of data points. Formally, an LSH function  $\mathcal{H}$  is defined for a metric space  $\mathcal{M} = (M,d)$ , where  $d$  is a distance metric in space  $M$ , with a given threshold  $T > 0$ , approximation factors  $f > 1$ , and probabilities  $P_{1}$  and  $P_{2}$ , where  $P_{1} \gg P_{2}$ .  $\mathcal{H}$  maps elements of the metric space to buckets  $b \in B$  and satisfies the following conditions for any two points  $q_{1}, q_{2} \in M$ : (1) If  $d(q_{1}, q_{2}) \leq T$ , then  $\mathcal{H}(q_{1}) = \mathcal{H}(q_{2})$  (i.e.,  $q_{1}$  and  $q_{2}$  collide in the same bucket  $b$ ) with probability at least  $P_{1}$ . (2) If  $d(q_{1}, q_{2}) \geq fT$ , then  $\mathcal{H}(q_{1}) = \mathcal{H}(q_{2})$  with probability at most  $P_{2}$ .
3.3 Building Block 2: Cost Function Design
Once we can estimate the coverage of an embedding space for a given user  $U$  as  $\tilde{\mathcal{E}}_f^U$ , we need to design a cost function  $\mathcal{C}:\mathbb{R}^{+}\to \mathbb{R}^{+}$  that maps from the estimated coverage to a cost. The cost function needs to be designed such that it does not significantly penalize legitimate users while imposing a severe penalty on adversaries to effectively prevent the encoder from being stolen. The semantics of the cost function's range depend on the type of costs that the defender wants to enforce. We discuss a broad range of options in Appendix C. These include monetary cost functions to adaptively charge users on a batch-query basis depending on their current coverage, costs in the form of additional computation that users need to perform in order to obtain their representations, similar to the proof of work in [18], costs in the form of delay in the interaction with the encoder behind the API [4], or costs in form of disk space that needs to be reserved by the user (similar to a proof of space [19, 20]). Which type of cost function is most adequate depends on the defender's objective and setup.
Exponential Cost Functions to Adjust Utility of Representations. In the concrete instantiation of B4B that we present in this work, we rely on costs in the form of the utility of the returned representations. We choose this concrete instantiation because it is intuitive, effective, and can be directly experimentally assessed. Moreover, it is even suitable for public APIs where, for example, no monetary costs are applicable. We adjust utility by adding Gaussian noise with different standard deviation  $\sigma$  to the returned representations. Since we do not want to penalize legitimate users with small coverage but make stealing for adversaries with growing coverage increasingly prohibitive, we instantiate an exponential cost function that maps from the fraction of hash buckets occupied by the user to a value for  $\sigma$ . We choose the general form of this function as
$$
f _ {\lambda , \alpha , \beta} \left(\tilde {\mathcal {E}} _ {f} ^ {U}\right) = \lambda \times \left(\exp^ {\ln \frac {\alpha}{\lambda} \times \tilde {\mathcal {E}} _ {f} ^ {U} \times \beta^ {- 1}} - 1\right) \tag {1}
$$
where  $\lambda < 1$  compresses the curve of the function to obtain low function values for small fractions of occupied buckets, and then we set a target penalty  $\alpha$  for our cost function at a specified fraction of filled buckets  $\beta$ . For instance, if we want to enforce a  $\sigma$  of 5 at  $90\%$  of filled buckets (i.e., for  $\tilde{\mathcal{E}}_f^U = 0.9$ ), we would need to set  $\alpha = 5$  and  $\beta = 0.9$ .
3.4 Building Block 3: Per-User Representation Transformations against Sybil Attacks
Given that our defense discourages users from querying with a wide variety of data points from different distributions, an adversary could create multiple fake user accounts (sybils) and query
different data subsets with more uniform representations from each of these accounts. By combining all the obtained representations and using them to train a stolen copy, the adversary could overcome the increased costs of stealing. To defend against such sybil attacks, we propose individually transforming the representations on a per-user level before returning them. As a result, the adversary would first have to map all the representations to one single unified space before being able to jointly leverage the representations from different accounts for their stolen copy.
Formally, for a given query  $q_{i}$ , the protected victim encoder produces a representation  $r_i = f_v(q_i)$  which is transformed by a user-specific transformation  $T_U(r_i)$  before being returned to the querying user  $U$ . For a new user  $U$ , the defense randomly selects the transformation  $T_U$  from all possible choices. Note that the randomness is also added on a per-transformation basis, instead of only on the level of selecting the transformations. For example, a permutation of the elements in the output representations should be different for each user.
We formulate two concrete requirements for the transformations. First, they should preserve utility for legitimate users on their downstream tasks, and second, they should be costly to reverse for an adversary.