SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also explore alternative methods to measure the distances between representations for queries sent to an API. One of them is to apply the cosine distance (where for two representations  $a$  and  $b$ , it is defined as:  $1 - \frac{a^T b}{||a||_2 \cdot ||b||_2}$ ) since it can be measured between individual data points in a pair-wise fashion. If the total pair-wise cosine distance between representations for a given user is small, then the user queries presumably come from a single downstream task distribution. Otherwise, a user might be malicious and would like to cover a large part of the representation space, then the total pair-wise cosine distance for the user's representations would be high. Note that in this case, the cosine distance can be replaced with any other distance measure, such as the Minkowski distance. We opt for the LSH in our reference implementation, since it is much less expensive to compute than cosine distance. LSH requires only  $2^{12} = 4096$  buckets that can be expressed as a binary table with the same number of elements, which requires in the worst case iterating over all of them to count how many are occupied. With more than 4096 queries sent by a given user, the computation on the LSH is sublinear  $< O(n)$  with respect to the number of user queries. For the cosine distance approach, the required computation grows quadratically  $O(n^2)$  with the number of queries.
C.2 Alternative Cost Functions
The cost functions can be designed from scratch manually or learned, for example, via an ML model, such as a neural network or SVM. In our initial version, the function was designed manually, where the underlying premise is that once a specified number of buckets is occupied, the cost should grow exponentially. Instead of defining such a function or providing the high-level parameters for functions that we contributed, one could learn an ML model that for a given number of buckets occupied, it should output an estimated cost, or even directly, the desired  $\sigma$  (standard deviation) of the noise added to representations. This method requires a relatively large number of data points to be provided for training the model, however, lowers the burden on a defender to either decide on the specific function or adjust its parameters. Thus, it could be more user-friendly, for example, not necessitating any mathematical background, but can be precise enough to obtain the desired behavior.
Note that instead of adding the calibrated noise (proportional to the estimated cost) to the representations, we could rather require a given user to pay a higher monetary cost for queries that cover a large fraction of the representation space, or force a user to solve a puzzle in a form of the proof-of-work [18], wait a specified amount of time via proof-of-Elapsed time (PoET) [4], or prove that
a specified amount of disk space was reserved [19, 20]. For example, consider the approach with PoET. A user sends queries to the API, which we cost based on their occupation of the embedding space. The user is sent a waiting time. The users' resource (e.g., a CPU) has to be occupied for this specific waiting time without performing any work. At the end of the specified amount of time, the user sends proof of the elapsed time, which can be easily verified by the server. PoET requires access to specialized hardware, for example, secure CPU instructions that are becoming widely available in consumer and enterprise processors. If a user does not own such hardware, proof of elapsed time could be produced using a service exposed by a cloud provider (e.g., Azure VMs that feature TEE 2). Note that if a server sends the time based on the calculated cost, the adversary might learn the cost function. Instead, the exact waiting time should be split in random subwaiting times and sent to the user one by one. Thus, a server should rather have a few rounds of exchange with the client to incur the additional cost.
C.3 Alternative to Transformations
Figure 6: Overview on Transformations. We depict the inner-workings of the transformations considered in this work.
As an alternative to the transformations used within this work (see Figure 6), one could use a different set of transformations or combinations thereof. The padding can be done with different constant values and combined with adding constant values within the representations. The padding and adding the constant values can be followed by shuffling the elements within the representations. We can apply the affine of binary transformations on top of the padding and shuffling. Additionally, we can also use other pre-defined linear transformations like rotations or shearing.
The representations could also be compressed to smaller vectors and the compression rate would depend on the occupation of the representation space, for example, the higher the number of occupied buckets in our hash table, the more compressed the output representations could be. Such representations could be compressed via FFT, a cosine transform, or standard compression techniques such as snappy [5]. If the information from the representations should not be lost, then the lossless compression techniques can be applied, for instance, zstd [6]. The only requirement of the compression techniques is to ensure that they do not decrease the accuracy on downstream tasks for legitimate users.
Another alternative is to incorporate an additional neural network layer for transforming the returned representations. The training of this supplementary layer should primarily focus on preserving the usability of the representations for legitimate users. This approach grants the API provider with additional capabilities, as it allows for the utilization of customized training objectives. For instance, if the API provider employs LSH (Locality-Sensitive Hashing) to estimate the coverage of the representation space, they can leverage buckets and train the additional layer to maintain high-quality representations exclusively for frequently-used buckets and their surrounding areas, while not prioritizing the rest of the representation space. This approach safeguards legitimate users
from any adverse effects, as their coverage of the representation space is minimal. Simultaneously, it ensures that adversaries are unable to exploit representations from the entire representation space.
D Sybil Attacks
We consider an adversary who generates  $n$  sybil accounts to steal the encoder from the API. For each of the accounts, the representations are transformed in a different way. Therefore, to replicate the victim model using all the obtained representations, the adversary has to map these representations into one single space. This can be done, for example, by training a neural network to perform the mapping.
We assume the adversary obtains  $\{N_1, N_2, \ldots, N_n\}$  many representations from the victim for each of the  $n$  sybil accounts. Without loss of generality, we assume the adversary maps them back to the embedding space of the first sybil account. To learn the mapping, the adversary can apply different strategies.
D.1 Sybil Strategies
We present three potential approaches that Sybils might want to apply to circumvent our defense. Consider three users:  $A$ ,  $B$ , and  $C$ , with their respective datasets  $D_A$ ,  $D_B$ , and  $D_C$ , each with different distributions to maximize extraction effectiveness. First, user  $A$  is selected to unify representations from other users  $B$  and  $C$ . User  $A$  would have to query from at least two different datasets  $D_B$  and  $D_C$ , while other users would act legitimately. Sybil attackers want to deploy as many users as possible but with more fake accounts, user  $A$  incurs high coverage of the representation space, and this is prevented by our single-user defense. In all other cases neither of the sybil users can act legitimately, thus they are already affected by the single-user defense. Second, user  $A$  would query from their own dataset  $D_A$  and partially from dataset  $D_B$ . Then user  $B$  would query from their own dataset  $D_B$  and partially from dataset  $D_C$ , and so on. This method is the most inconspicuous but requires a number of remappings that scales super-exponentially with the number of fake accounts, which is impractical. Finally, each user would query from their respective dataset, for example, user  $B$  would query from dataset  $D_B$  and additionally from a remapping dataset, e.g.,  $D_A$ . Representations could be unified by mapping them to  $A$ 's representations. The last approach as well as all other cases reduce to the minimum of remapping between representations of a pair of users. We show that our defense cuts such attempts short by ensuring that the remapping between representations is prohibitive even for a pair of users.
E Additional Related Work
One of the main workhorse techniques used in the encoders is contrastive learning, where the representations are trained so that the positive pairs (two augmented versions of the same image) have similar representations while negative pairs (augmentations of two different images) have representations which are far apart.
SimSiam utilizes Siamese networks (two encoders with shared weights) but with a simplified training process and architecture. In contrast to the previous frameworks, such as SimCLR [10], SimSiam's authors show that negative samples are unnecessary and collapsing solutions can be avoided by applying the projection head to one of the encoders, and a stop-gradient operation to the other. SimSiam minimizes the negative cosine similarity between two randomly augmented views of the same image from the Siamese encoders, which is expressed via a symmetrized loss [22]. This creates a simple yet highly effective representation learning method.
DINO is another popular representation learning framework. While SimSiam uses CNNs, DINO employs vision transformers (ViTs). It trains a student and teacher encoder with the same architecture, updating the teacher with an (exponential moving) average of the student. Different random transformations of the same image are passed through both encoders. The student receives smaller image crops, forcing it to generate representations restoring parts of the original image. The training objective is minimizing cross-entropy loss between teacher and student representations.
F.1 Details on Experimental Setup
The end-to-end experiments on stealing SimSiam and ViT DINO encoders were done using 3 A100 GPUs. Detailed experiments including mapping, transformations and the evaluation was performed using a single computer equipped with two Nvidia RTX 2080 Ti GPUs.
F.2 Datasets Used
CIFAR10 [28]: The CIFAR10 dataset consists of 32x32 colored images with 10 classes. There are 50000 training images and 10000 test images.
SVHN [32]: The SVHN dataset contains 32x32 coloured images with 10 classes. There are roughly 73000 training images, 26000 test images and 530000 "extra" images.
ImageNet[15]: Larger sized coloured images with 1000 classes. As is commonly done, we resize all images to be of size  $224 \times 224$ . There are approximately 1 million training images and 50000 test images.
STL10 [13]: The STL10 dataset contains 96x96 coloured images with 10 classes. There are 5000 training images, 8000 test images, and 100000 unlabeled images.
LAION-5B [35] The LAION-5B dataset consists of 5,85 billion CLIP-filtered image-text pairs. The dataset was crawled from publically available internet.
F.3 More Results for the End2End Empirical Evaluation
We consider fine-tuning parameters  $\beta$ ,  $\lambda$ , and  $\alpha$  for our cost function and the intuitive meaning behind these parameters. In general, our recommendation is to adjust the parameter  $\beta$  that specifies how many buckets are allowed to be filled by users' downstream tasks. On the other hand, when parameter  $\lambda$  is increased, this causes a higher amount of added noise before we reach the number of buckets specified by  $\beta$ , which lowers the performance of a given downstream task relatively early. For example, a higher value of  $\lambda$  in Figure 4, would cause an increase in the amount of added noise much earlier than for the target value of  $\beta$ . Finally, parameter  $\alpha$  controls the amount of noise once the number of buckets specified by  $\beta$  is reached. Thus, in Figure 4, we set  $\alpha = 1$  and the maximum standard deviation of the added Gaussian noise is 1.
(a) Cost Function for different  $\lambda$  parameter values.
(b) Cost Function for different  $\alpha$  parameter values.
Figure 7: Effects of  $\lambda$  and  $\alpha$  parameters on the Cost Function. We present the Cost Function for  $\alpha = 1$ ,  $\beta = 80$  and different values of  $\lambda$  (left) and  $\lambda = 10^{-6}$ ,  $\beta = 80$  and different values of  $\alpha$  (right).
Table 2: Stealing and Using Encoders With and Without our Defense. The model used in the experiments is Simsiam, with the following parameters for the cost function  $\lambda = 10^{-4}$ ,  $\alpha = 1$ , and  $\beta = 80\%$ , and the number of buckets equal to  $2^{12}$ . Due to the higher value of the parameter  $\lambda$ , we observe lower performance on downstream tasks for the attackers since the magnitude of noise added to the representations is higher. However, for more complicated tasks than CIFAR10, this change might cause a potential drop in accuracy for the legitimate users.
USER	DEFENSE	# QUERIES	DATASET	TYPE	CIFAR10	STL10	SVHN	F-MNIST
LEGIT	NONE	ALL	TASK	QUERY	90.41±0.02	95.08±0.13	75.47±0.04	91.22±0.11
LEGIT	B4B	ALL	TASK	QUERY	90.02 ±0.1	94.88 ±0.17	74.72 ±0.13	91.76 ±0.09
ATTACK	NONE	50K	IMGNET	STEAL	65.2 ±0.03	64.9 ±0.01	62.1 ±0.01	88.5 ±0.01
ATTACK	B4B	50K	IMGNET	STEAL	28.22 ±0.04	26.62 ±0.02	19.62 ±0.02	78.41 ±0.01
ATTACK	NONE	100K	IMGNET	STEAL	68.1 ±0.03	63.1 ±0.01	61.5 ±0.01	89.0 ±0.07
ATTACK	B4B	100K	IMGNET	STEAL	17.73 ±0.18	15.59 ±0.61	19.53±0.01	55.11 ±0.05
SYBIL	B4B	50K+50K	IMGNET	STEAL	33.43 ±0.03	31.18 ±0.12	22.91 ±0.01	75.35 ±0.05
Table 3: Stealing and Using Encoders With and Without our Defense. The model used in the experiments is Simsiam, with the following parameters for the cost function  $\lambda = 10^{-6}$ ,  $\alpha = 1$ , and  $\beta = 50\%$ , and the number of buckets equal to  $2^{12}$ . This experiment corresponds to considering  $50\%$  of buckets filled as a too-large coverage of the embedding space. This improves the defense but again might potentially harm the performance of more complicated tasks than CIFAR10 since they could occupy more buckets than  $50\%$ .
USER	DEFENSE	# QUERIES	DATASET	TYPE	CIFAR10	STL10	SVHN	F-MNIST
LEGIT	NONE	ALL	TASK	QUERY	90.41±0.02	95.08±0.13	75.47±0.04	91.22 ±0.11
LEGIT	B4B	ALL	TASK	QUERY	90.27 ±0.07	95.12 ±0.13	74.94 ±0.16	91.66 ±0.05
ATTACK	NONE	50K	IMGNET	STEAL	65.2 ±0.03	64.9 ±0.01	62.1 ±0.01	88.5 ±0.01
ATTACK	B4B	50K	IMGNET	STEAL	15.52 ±0.37	12.57 ±0.23	19.53±0.01	23.17 ±0.01
ATTACK	NONE	100K	IMGNET	STEAL	68.1 ±0.03	63.1 ±0.01	61.5 ±0.01	89.0 ±0.07
ATTACK	B4B	100K	IMGNET	STEAL	16.27 ±0.04	13.93 ±0.35	19.54 ±0.02	54.69 ±0.02
SYBIL	B4B	50K+50K	IMGNET	STEAL	30.14 ±0.01	29.57 ±0.08	19.99 ±0.03	71.72 ±0.01
Table 4: Stealing and Using Encoders With and Without our Defense. The model used in the experiments is Simsiam, with the following parameters for the cost function  $\lambda = 10^{-6}$ ,  $\alpha = 1$ , and  $\beta = 30\%$ , and the number of buckets equal to  $2^{12}$ . Since the value of parameter  $\beta$  is decreased substantially to  $30\%$ , we observe a drop in accuracy for legitimate users. For example, more than  $1\%$  for CIFAR10. In the next Table 5, we show that by also decreasing the parameter  $\alpha$ , we can attenuate this harmful effect and retain higher accuracy for legitimate users. In case of an attack, for 100k stealing queries, we observe much lower accuracy levels than for  $\beta = 50\%$  shown in Table 3.
USER	DEFENSE	# QUERIES	DATASET	TYPE	CIFAR10	STL10	SVHN	F-MNIST
LEGIT	NONE	ALL	TASK	QUERY	90.41±0.02	95.08±0.13	75.47±0.04	91.22 ±0.11
LEGIT	B4B	ALL	TASK	QUERY	88.1 ±0.11	94.92 ±0.11	74.37 ±0.02	91.67 ±0.07
ATTACK	NONE	50K	IMGNET	STEAL	65.2 ±0.03	64.9 ±0.01	62.1 ±0.01	88.5 ±0.01
ATTACK	B4B	50K	IMGNET	STEAL	30.82 ±0.09	26.37 ±0.07	21.87 ±0.03	66.0 ±0.02
ATTACK	NONE	100K	IMGNET	STEAL	68.1 ±0.03	63.1 ±0.01	61.5 ±0.01	89.0 ±0.07
ATTACK	B4B	100K	IMGNET	STEAL	9.57 ±0.17	9.83 ±0.09	19.57 ±0.01	27.06 ±0.46
SYBIL	B4B	50K+50K	IMGNET	STEAL	29.15 ±0.02	28.67 ±0.06	19.98 ±0.03	70.62 ±0.03
Table 5: Stealing and Using Encoders With and Without our Defense. The model used in the experiments is Simsiam, with the following parameters for the cost function  $\lambda = 10^{-6}$ ,  $\alpha = 0.1$ , and  $\beta = 30\%$ , and the number of buckets equal to  $2^{12}$ . Due to the lower performance on downstream tasks observed in Table 4 while keeping the parameter  $\beta$  fixed to  $30\%$  and  $\lambda$  fixed to  $10^{-6}$ , we decrease the value of parameter  $\alpha$  to 0.1, which increases the performance of legitimate users on their downstream tasks. In this experiment, we also carry out a sybil attack with more accounts (4 instead of 2), but observe that this modification does not improve the performance of the attacker. With more accounts, a sybil has to sacrifice more queries for the remappings between the representations from different accounts. Additionally, note that each account introduces a different remapping error by the mint of different transformations applied to each account by B4B.
USER	DEFENSE	# QUERIES	DATASET	TYPE	CIFAR10	STL10	SVHN	F-MNIST
LEGIT	NONE	ALL	TASK	QUERY	90.41±0.02	95.08±0.13	75.47±0.04	91.22±0.11
LEGIT	B4B	50K	CIFAR10	QUERY	90.17 ±0.1	94.92 ±0.09	74.97 ±0.13	91.71 ±0.08
ATTACK	NONE	50K	IMGNET	STEAL	65.2 ±0.03	64.9 ±0.01	62.1 ±0.01	88.5 ±0.01
ATTACK	B4B	50K	IMGNET	STEAL	19.95 ±0.19	15.54 ±0.34	19.57 ±0.01	23.50 ±0.19
ATTACK	NONE	100K	IMGNET	STEAL	68.1 ±0.03	63.1 ±0.01	61.5 ±0.01	89.0 ±0.07
ATTACK	B4B	100K	IMGNET	STEAL	10.35 ±0.19	12.37 ±0.69	19.34 ±0.01	68.93 ±0.17
SYBIL	B4B	4×25K	IMGNET	STEAL	33.15 ±0.04	30.23 ±0.07	20.87 ±0.01	72.19 ±0.02
F.4 B4B vs Static Noise Addition Defenses
We compare our B4B against the current state-of-the-art baseline defense, namely adding a static addition of noise to all the returned representations (as proposed in [16] (Section A.4),[29, 36]). For the Table 6, we use the same setup as in Table 1 (with an ImageNet pre-trained encoder).
Our results show the following insights:
1. If the amount of noise is small ( $\sigma = 0.1$ ) then the performance drop is negligible but for both a legitimate user (row 2) and an adversary (row 6). In this case, the defense does not affect the adversary at all (compare rows 5 & 6).
2. If the amount of noise is large ( $\sigma = 10$ ) then the performance drop is large for both a legitimate user (row 3) and an adversary (row 7). In this case, the encoder is worthless for legitimate users since the performance is too low.
Table 6: Stealing and Using Encoders with Static Noise Addition Defenses vs. Our B4B Defense. Adding a small amount of noise results in negligible drop in performance for both legitimate user (row 2) and an adversary (row 6). Adding a large amount of noise defend stealing (row 7), but significantly harm legitimate users at the same time (row 3). Our B4B defense solves the above problem and provides high performance for legitimate users (row 4) while effectively defending the encoder against stealing attacks (row 8).
USER	DEFENSE	# QUERIES	DATASET	TYPE	CIFAR10	STL10	SVHN	F-MNIST
LEGIT	NONE	ALL	TASK	QUERY	90.41 ±0.02	95.08±0.13	75.47±0.04	91.22±0.11
LEGIT	NOISE σ=0.1	ALL	TASK	QUERY	90.20±0.03	95.15±0.13	75.29±0.09	91.24±0.02
LEGIT	NOISE σ=10	ALL	TASK	QUERY	65.11±0.45	76.37±0.14	33.23±0.09	65.83±0.13
LEGIT	B4B	ALL	TASK	QUERY	90.24±0.11	95.05±0.1	74.96±0.13	91.7±0.01
ATTACK	NONE	50K	IMGNET	STEAL	65.2±0.03	64.9±0.01	63.1±0.01	88.5 ±0.01
ATTACK	NOISE σ=0.1	50K	IMGNET	STEAL	64.92±0.04	64.61±0.02	62.35±0.01	88.41±0.01
ATTACK	NOISE σ=10	50K	IMGNET	STEAL	36.32±0.2	32.59±0.06	20.59±0.01	74.94±0.02
ATTACK	B4B	50K	IMGNET	STEAL	35.72±0.04	31.54±0.02	19.74±0.02	70.01±0.01
F.5 Additional Embedding Space Coverage experiments
We present additional experiments on measuring the coverage of the representation space.
First, we use the same set-up as from Table 1 - SimSiam with ResNet50 pretrained on ImageNet. When querying the encoder with ImageNet-Full (includes all 1000 classes) and LAION-5B datasets, they both occupy a large fraction of the representation space of the victim encoder, as shown on Figure 8. In contrast, CIFAR10 covers the smallest portion of the representation space as the simplest dataset tested. ImageNet-Dogs (with only 118 classes for dog breeds) falls in the middle, occupying more space than CIFAR10 but less than ImageNet-Full and LAION-5B. Its intermediate coverage aligns with its mid-level difficulty compared to the other datasets. As indicated by representation space coverage, stealing the encoder is similarly effective with ImageNet-Full and LAION-5B datasets, as both datasets cover a large fraction of the representation space. Overall, Figure 8 demonstrates that: 1) our B4B can successfully protect the encoder model even from attackers stealing with data that was not used to train the model (LAION-5B in this case) and 2) while providing clean representation for users querying from downstream tasks that are part of more complicated datasets (ImageNet-Dogs).
Figure 8: Fraction of Occupied Buckets (Embedding Space Coverage) for the ImageNet encoder. Representations for the downstream datasets (CIFAR10, ImageNet - Dogs) occupy a smaller fraction of buckets than representations from the complex ImageNet or LAION-5B datasets. The underlying encoder is SimSiam pre-trained on ImageNet with ResNet50.
Our method of measuring the embedding space coverage is not limited to a particular encoder or dataset used for pretraining. We demonstrate this in Figure 9, showing the fraction of occupied buckets for a SimCLR [11] Resnet34 encoder pretrained on CIFAR10.
Figure 9: Fraction of Occupied Buckets (Embedding Space Coverage) for the CIFAR10 encoder. B4B can be applied to an encoder trained on CIFAR10. Representations for the downstream datasets (FashionMNIST, SVHN) occupy a smaller fraction of buckets than representations from CIFAR10, CIFAR100, and STL10 datasets. The underlying encoder is SimCLR pre-trained on CIFAR10 with ResNet34.
F.6 Setting the number of buckets
We present our procedure to find an optimal number of buckets in Figure 10.
(a) Number of buckets  $= 2^{8}$
(b) Number of buckets  $= 2^{10}$
(c) Number of buckets  $= 2^{12}$
(d) Number of buckets  $= 2^{14}$
Figure 10: Estimating Embedding Space Coverage through LSH on the SimSiam Encoder. We extend the results from Figure 3(a) and present the fraction of buckets occupied by representations of different datasets as a function of the number of queries posed to the encoder. We consider different number of buckets in the LSH table. We observe that  $2^{8}$  buckets is to small since queries from the ImageNet dataset saturate all the buckets after around 50k queries, while the number  $2^{14}$  of buckets is too large since it is never occupied more than  $40\%$ . Thus, the number  $2^{12}$  buckets is a good middle ground. Subfigure (c) corresponds to Figure 3 from the main paper. We also use the same notation and carry out our experiments in the same way as in Figure 3.
F.7 Results for DINO
We show that our defense is also applicable to the DINO encoder. The occupation of the representations space is presented visually in Figure 11. We also show that the number of buckets  $2^{12}$  is optimal for DINO in Figure 12. The impact of transformation on the representations from DINO is shown Table 8. Finally, the end to end experiment for DINO is presented in Table 7.
Figure 11: Representations from Different Tasks Occupy Different Sub-Spaces of the Embedding Space. Presented for FashionMNIST, SVHN, CIFAR10, and STL10. In this plot, we used the DINO ViT Small encoder trained on ImageNet.
Figure 12: Estimating Embedding Space Coverage through LSH on the DINO Encoder. The number of buckets is set to  $2^{12}$ . We also use the same notation and carry out our experiments in the same way as in Figure 3.