Joint Attribute and Model Generalization Learning for Privacy-Preserving Action Recognition
Authors: Duo Peng
Li Xu
Qiuhong Ke
Ping Hu
Jun Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Privacy Preservation
Action Recognition
Meta-Learning

Abstract: Privacy-Preserving Action Recognition (PPAR) aims to transform raw videos into anonymous ones to prevent privacy leakage while maintaining action clues, which is an increasingly important problem in intelligent vision applications. Despite recent efforts in this task, it is still challenging to deal with novel privacy attributes and novel privacy attack models that are unavailable during the training phase. In this paper, from the perspective of meta-learning (learning to learn), we propose a novel Meta Privacy-Preserving Action Recognition (MPPAR) framework to improve both generalization abilities above (i.e., generalize to *novel privacy attributes* and *novel privacy attack models*) in a unified manner. Concretely, we simulate train/test task shifts by constructing disjoint support/query sets w.r.t. privacy attributes or attack models. Then, a virtual training and testing scheme is applied based on support/query sets to provide feedback to optimize the model's learning toward better generalization. Extensive experiments demonstrate the effectiveness and generalization of the proposed framework compared to state-of-the-arts.

Joint Attribute and Model Generalization Learning for Privacy-Preserving Action Recognition
Duo Peng
Xu
SUTD
SUTD
Singapore
Singapore
duo who__enn@@mmaii...mmmm sutd. edu.sg
1i li_xu@mymail wttpl/ammallsuu.....mmmmmmmm wut.d.euu..... edu. sg
Qiuhong Ke Monash University
Ping Hu UESTC
Jun Liu" SUTD
Australia Qiuhong Ke@monash. edu
China
Singapore
chinahuping@@mail.. com
Lun_liu@sudd.cm edu. sg
Abstract
Privac-Prreserving Action Recognition (PPAR) aims to transform videos into anonymous ones to prevent privacy leakage while maintaining action clues, which is increasingly important problem in intelligent vision applications. Despite recent efforts this task, it is still challenging to deal with novel privacy attributes and novel privacy attack models that unavailable during the training phase. In this paper, from the perspective of meta-learning (learning to learn), we propose novel Meta Privacy-Preserving Action Recognition (MPPAR) framework improve both generalization abilities above (i.e., generalize to novel privacy attributes and novel privacy attack models) in unified manner. Concretely, we simulate train/test task shifts by constructing disjoint support/query sets w.r.t. privacy attributes or attack models. Then, virtual training and testing scheme is applied based on support/query sets to provide feedback to optimize the model's learning towards better generalization. Extensive experiments demonstrate the effectiveness and generalization of the proposed framework compared to saate-of-the-arre
Introduction
Recently, smart home cameras such as Amazon Echo and Google Nest Cam, have been widely used in millions of families to provide intelligent monitoring services to enhance security (c.g.. detecting unusual behaviors and alerting householders remotely) [1, 2]. However, the widespread use of such cameras has raised privacy concerns and even pressured changes in the regulations/laws in Europe visual data that often contains rich privacy information (such as face, gender and skin color, etc) to the intelligent system deployed on cloud or public servers for analysis, which leads to the risk of privacy leakage [3, 4]. While traditional cryptographic solutions can provide video encryption during transmission between the local camera and the intelligent system, they would struggle with preventing authorized agents (e.g.. the system administrator) from misusing the privacy information [13, 14].
and US [13], as generally requires uploading ddvvcce-capture
Therefore, there is an urgent need to find an appropriate anonymization transformation crase privacy information from the captured raw visual data at the local camera end (before the upload), while still enabling certain target tasks required by the intelligent system [5, 6, 7]. Meanwhile, since action recognition is a fundamental video understanding task with wide applications (such as the dntove-mentionsl smart home cameras), there is growing interest in studying anonymization transformation for privacy preservation in video action recognition, i.e., Privacy-Preservicy Action
"CoresponningAuthor
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Recognition (PPAR) [8, 9, 10, 11, 12, 13, 14, 15]. The goal of PPAR is to train an anonymization model to transform the captured raw videos, so that the transformed (anonymous) videos not only avoid undesired privacy disclosure but also enable performing the specified action recognition task.
Not that while using sreleton-based models could naturally handle the privacy concerns in action recognition, the research regarding privacy preservation in RGB-based models still remains important and practical. Here are the reasons: In many scenarios, it is sometimes also necessary to incorporate information beyond action into the action recognition model. For instance, in the kitchen scenario [18] which contains many fine-gr anneggrined actions, there are many actions that can be similar, such as wash dishes and cut vegetables (both require one hand to hold onto the object while the other hand performs periodic shaking motions). In this case, accurately recognizing the action requires richer contextual information from the background and surroundings. In such a scenario, simply using the skeleton to remove all other information will lead to performance degradation [19]. This indicates that in many scenarios, we still require RGB videos to provide reliable action recognition. Therefore, In RGB-based action recogtion, learning how to remove privacy information while retaining both action information and related visual cues for accurate action recognition becomes extremely important. Recently, it has attracted increasing attention and a series of works [13, 15, 28] has been proposed, reflecting the significance of tackling privacy concerns within RGB-based action recognition. Furthermore, RGB-base action recognition has already been extensively deployed in many household applications [20], such Amazon Echo and Google Nest Cam. Therefore, the privacy preservation for RGB-basce action recognition (i.c., PPAR) is of utmost urgency.
In PPAR, some video downsampling- based methods [8, 9, 10] propose to produce extremely low- resolution videos to create -rreeeei-r ""anoymmous videos". Besides, there exist some obfuscation-based methods [11, 12] that propose to produce "anonymous videos" by blurring videos. Although these video processing- visual content without end-to-end learning often cannot provide a good trade-off between action recognition and privacy preservation. To handle this issue, recent methods [13, 14, 15] propose to explicitly optimize the action- -ativn-privacy trade-off by training an anonymization model to remove privacy information through an adversarial learning framework. Specifically, the adversarial learning framework incorporates the action recognition model and the privacy classification model. to train the anonymization model using a minimax optimization strategy where the action recognition loss is minimized while the privacy classification loss is maximized, ultimately achieving good trade-off between action recognition and privacy preservation. However, despite the progress, these methods still struggle in real-world applications due to their limited generalization ability, which can manifest in the following two aspects:
methods have shown promising results, simply discarding
In real-world applications, video data often contains very rich personal information that are difficult to be exhaustively enumerated, such as palm prints, fingerprints, faces, credit cards, cellphone screen information, etc. This makes it almost impossible to ensure that all potential privacy information has been comprehensively labeled in the training data. Existing works focusing on locating and removing privacy attributes through labeled supervision do not explicitly deal with the novel (unseen) privacy attributes, which can be unsuitable to handle the real-world applications where the video data can include broad range of potential privacy attributes.
framework should also be model-aggostic i.e., preventing various possible privacy attack (i.e., privacy classification) models from stealing privacy information [13, 14). However, previous studies often use specified privacy attack models for privacy-preserving training, which might limit their privacy-preserving performance when facing novel privacy attack models. Furthermore, due to the rapid development and evolution of privacy classification models [16. 17]. it is impractical to collect all possible privacy attack models for training to ensure the attack model during testing is always seen before.
-On the other hand, an ideal
Thus in this paper, we aim to build ter generalization abilities in terms of both the mentioned aspects: handling novel privacy attributes as well as novel privacy attack models. We observe that these two aspects both require the anonymization model to learn generalizable knowledge about identifying and removing privacy information, that can generalize beyond the given dataset (with limited labeled privacy attributes) and specified attack models. Despite the conceptual simplicity, how to guide anonymization models to learn such generalizable knowledge is challenging. Here from the perspective of meta Iearn-
presevypi action recognition framework with bet-
2

---

ing, we propose novel framework, Meta Frivacy-Preserving Action Recognition (MPPAR), to simultaneously improve both generalization abilities of the anonymization model.
Meta learning, also known as learning to learn, aims to enhance the model generalization capability by performing virtual testing during model training [21, 22]. Inspired by this, to improve the generalization capability of the anonymization model, our framework incorporates a virtual training and testing scheme that consists of three steps: virtual training, virtual testing and meta optimization. Specifically, we first construct a support set for virtual training, and a query set for virtual testing. Since we aim to improve the model generalization capability w.r.t. both novel privacy attributes and novel privacy attack models, we construct the support set and query set accordingly as follows. For handling novel privacy attributes, we split the training data to construct a support set and query set where the videos of the query set contain labeled privacy attributes that are unseen in the support set. Similarly, for handling novel privacy attack models, we split a collection of privacy attack models (for training) into support set and a query set, which contain different attack models.
Based on the constructed sets, we design novel virtual training and testing scheme. Specifically, we first train the model over the support set (i.e., virtual training), and then test the trained model on the query set (i.e., virtual testing). Based on the testing performance (loss) on the query set, we perform the meta optimization to update the model for better generalization capability. Since the query set contains novel privacy attributes (and novel attack models) w.r.t. the support set, by improving the model's 'ss testing performance on the query set after training on the support set via virtual training and testing, the model is driven learn more generalizable knowledge that can help remove potentially unseen privacy attributes (and defend against unknown privacy attackers), during virtual training. In this way, we can effectively improve the model generalization capability, even when handling samples with novel privacy attributes and facing attacks from novel privacy attack models.
2 Related Work
DriracyPrreservigg. Action Recognition (PPAR). Existing PPAR methods can be broadly classified
into three main categories: (1) downsampling-baseee methods, (2) cefuscattoon-bsee methods, and (3) adversarial-lcamingg -eallrrinnnn-ueeed based methods. Generally, downsampling based methods [8, 9, 10] propose to doon-sample each video frame into a low-resolution frame to anonymize the privacy information. Obfuscation-bast methods [11, 12] mainly use off-the-shel object detectors to first detect privacy regions, and then blur the detected regions. Recently, in order to achieve better action- certiooy--vvcy trade- off, adderssrria--lariing-bass
methods [13, 14, 15, 28] adopt minimax optimization strategy (i.e., adversarial learning) to simultaneously optimize the action recognition and the privacy preservation. Specifically, Sudhakar et al. [15] proposed a BDQ anonymization model to preserve privacy in a spatio-temporal manner. While effective in dealing with seen privacy attributes and attack models, this method may struggle in handling the generalization problem (c.g.. novel attributes or novel attack models). Although Wu et al. [13, 14] have proposed a model-ensemble strategy to enhance the model generalization ability to novel attack models, such a strategy might make the trained anonymization model biased towards the attack models used for training. Dave et al. [28] proposed a novel contrastive learning approach to train a visual encoder for sitribute-generalin preservation. Nevertheless, due its focus on generalization learning without utilizing labeled data, it may lead to sub-optimal privacy protection performance when compared to supervised methods on the labeled attributes. Differently, in this paper, we propose novel framework that not only retains the benefits from learning with labeled data, but also simultaneously improves the privacy-preserve performance on both novel privacy attributes and novel privacy attack models, in a unified manner.
privacy
Meta Learning. The paradigm of learning to learn, known as meta learning. has emerged primarily to tackle the problem of few-shot learning [21, 22, 22, 23, 24J. Specifically, MAML [21] and its subsequent works [22, 23] aim to learn a good initialization of network parameters to achieve fast test-time updates and adapt to new few-shot learning tasks. More recently, meta learning has also been explored in other domains [255. 26, 27] to enhance model performance without requiring test-time updates. Different from typical meta learning methods for few-shot learning, our method aims to address the challenging PPAR problem involving novel attributes and unknown attack models. To this end, we propose novel framework to optimize the PPAR model via a virtual training and testing scheme over the carefully constructed support and query sets.
3

---

3
min Basic Model
Here we first introduce an sntersarilllleamnnng based PPAR model and its training strategy, which has shown promising performance and served as the basis of many state-of-the-an art methods [13, 14, 15]. In this paper, we also adopt this model as the basis of our framework. In this basic model, the original video data X is sent into the anonymization model fD to generate the anonymous video (X).. To train the anonymization model f its output (i.e., the anonymized video (X)) is sent into an action recognition model ʃ and privacy classification (attack) model S respectively. The action recognition model predicts the class of the action presented the anonymized video, while the privacy classification model predicts the existence of cach related privacy attribute in the video. The three models (f fA, and f are all learnable neural networks, and are trained under an adversarial learning algorithm. Specifically, the optimization objective for updating the anonymization model fD is defined as:
min L, where: L= Laction (( VA) pliprieacy (S (X)())) YP), (I) fD
where Y denotes the action recognition label, Luction denotes the classification (cross-entropy) loss. Y represents privacy attribute labels, Lprivacy is the multiple binary cross-entropy loss for multiple privacy labels and is a weight coefficient to balance these two terms. By minimizing Laction and -privacy (i.e., maximizing Lprivacy). the anonymization model fo optimized learn to retain the video content that can maintain the action recognition accuracy while removing privacy attribute information much as possible to lower the privacy classification accuracy, thus to fulfill the goal of privacy-preserve action recognition.
The optimization objective for updating the action recognition model f is defined as:
Lactson (I^ (X)),
f
The optimization objective for updating the privacy classification model f is defined as:
min I Teprivacy (J P (X)()))
Eq. 2 and 3 are used to maintain the reliable performance of the action recognition model and the privacy classification model, respectively. The total learning algorithm is handled as competition (adversarial) game [13, 14] among Ey. 1, 2 and 3. Thus after competition with each other, the capabilities of all three models (i.c., foh fa and fr) can be gradually enhanced.
Proposed Framework
In PPAR, existing state-of the-art mod- els: (1) generally focus on removing privacy information based on given pri- vacy attribute labels seen by the model during training, and (2) typically han- dle the privacy preservation task by fooling the predefined privacy classi- fication (attack) models. As result, after training with the given attribute labels and attack models, these meth- ods often struggle when encountering new privacy attributes or new attack models that are unseen during train- ing, To handle this issue, we aim to guide the model to learn more general- izable knowledge that can be applied for privacy preservation an attribute- agnostic and model-agnostic manner, which however is non-trivial. Here from the perspective of meta learning, we propose novel MPPAR framework to improve generalization abilities of the anonymization model w.r.t. the above two aspects in unified manner.
-i-- Allack Modk: Mekileett
ctual update (Fg
Arriboee luur oalor
Support Set D,
Anonymizatian Model /
Bg
Virtual Training Loss Luy
Virtual update (Bg
"Novel Model: egg, ResNir
"Noval 'A ttmbbbte eg. muaity
Query Set
Virrual Testing Loss
Intermediazy
Figure 1: Illustatioon of our virtual training and testing scheme. This scheme is handled by three steps: Virtual Training (marked in blue), Virtual Testing (green), and Meta Optimization (red).

---

As mentioned before, our framework is based on the Basic Model that contains three training objectives for updating fu, f and fP. respectively. Since we aim to improve f" for better generalization, in our framework, we only modify the training of fe (Eq. 1), while the updates of f (Eq. 2) and fP (Eq. 3) remain the same. More specifically, our framework incorporates virtual training and testing scheme for training fd. In this scheme, taking the procedures for handling novel privacy attributes as an instance, we intentionally split the original training set to construct support set Da. and query set Da where D, contains novel privacy attribute labels w.r.t. D,. Then, we first train the anonymization model f using the support set D, (virtual training), and then test the model's performance on the query set D, (Virtual testing). As the query set contains novel privacy attributes w.r.t. the support set, if the model can still achieve good testing performance the query set after being trained on the support set, it indicates that the trained model has learned more generalizable knowledge about removing various potential privacy information (including the novel privacy attributes) that are irrelevant to action recognition. With the virtual testing performance as feedback, we can optimize the virtual training process to drive the model's ' learning behavior towards learning more attribute- atrribute-ennrralizable generalizable knowledge. Via similar process, we can also optimize the model handle novel privacy attack models. Below, first describe the virtual training and lesting scheme, and then discuss how we construct the support set and query set.
4.1 Virtual Training and Testing
Virtual Training. As shown in Fig. (blue), during virtual training, we train the anonymization model via conventional gradient descent with the support set D,. Specifically, we denote the parameters of the anonymization model as Φ and calculate the virtual training loss Ty tr as:
L = ((o.D))
(4)
where L is the loss function (defined in Eq. 1) for updating the anonymization model. After calculating the virtual training loss, we can update our anonymization model's parameters e via gradient descent:
(5) where is the learning rate for virtual training. In this way, the model may be easy to learn the knowledge specific to the support set D. (e.g., removing privaey attributes labeled the support set D.). However, it might not be beneficial for the model performance on the query set (e.g.. the query set contains novel privacy attributes). Therefore, in this step, we do not actually update the amonymization model to be ø' (hence the term "virtual"),. Instead, the virtually updated model parameters ø' merely serve as an intermediary to calculate the virtual testing loss Le which is described in the next step.
--= (r)
Virtual Testing. As shown in Fig. 1 (green), after virtual training, we then evaluate the performance of the virtually trained model on the query set Dg:
(o)e = L,)
(6)
As we intentionally make the query set contain novel attributes w.r.t the support set, this virtual testing loss Lo te te()") can indicate the model's generalization capability to handle the privacy attributes beyond the support set after virtual training. The lower the virtual testing loss is, the better the model (after training) generalizes unseen attributes. Thus, to(")) can serve as a feedbuck to drive the model to adjust its training on the support set towards learning more generalizable knowledge via the following meta optimization step.
Meta Optimization. As discussed above, we aim to optimize the initial model o so that after learning (update) on the support set (i.e., & -> σ'), it can also obtain good testing performance (i.e., low L te (Ø')) on the query set that contain novel privacy attributes against the training data. To this end, we draw inspirations from MAML [21] and formulate the meta optimization objective as:
min Ly (0) + Le
min (x) + L (0 a aVoo
where the first term Lr (Ø) indicates the model's training performance, and the second term Top te(b") represents the model's testing performance (on novel privacy attributes) after virtual updating. Note that instead of optimizing o and o' sequentially, the meta optimization mentioned
5

---

above is performed solely on the initial model é, while ø' merely serves as an intermediary for evaluating the model's generalization performance on novel privacy attributes. Intuitively, the objective in Eq. 7 is to learn how to train & on the support set (i.e., lower t (+)) for better generalization performance (i.e., lower Tog te(")) Based on Eq. 7, we actually update the model with respect to @ as:
0 - の VV (L (0) + Lo
a (V
(8)
where 3 denotes the learning rate for meta-optimization. The above process of meta optimization
is illustrated in Fig. 1 (red). As both Lu (r))) and (e)) in Eq. 8 are based on loss L(.) (Eq. 1), which is proposed to learn to keep the action recognition accuracy while removing privacy attribute information, thus by learning the meta optimization updating rule in Eq. 8, the anonymization model can be guided to learn more generalizable knowledge about retaining action clues while removing other irrelevant visual content that includes various potentially unseen privacy attributes.
Here we provide an intuitive explanation of the meta optimization. In the virtual training step, we simply learn a model on the support set (i.c., ø + o'). The model o' captures knowledge of the provided set well, yet may suffer from the limited generalization ability when handling novel attributes. Hence, is critical to adjust the training process to optimize the model in a less biased way. To this end, we add virtual testing loss term shown in Eq. 7. Since we intentionally design the support set and query set to be disjoint, the virtual testing loss naturally serves as a regularization (feedback) to penalize the learning objective with second-order gradients (i.e., meta gradients, see Eq. 8) when the model generalizes poorly after virtual training. As a result, the biases of the training set are suppressed and the model turns to learn more generalizable knowledge that can help remove as much action-irrelevan information (including various potential privacy attributes) as possible. Note again, while we illustrate the above viriual training and lesting scheme using privacy attributes as an example, this scheme is equally applicable for handling novel privacy attack models. We will discuss how to simultancously address both aspects in Sec. 4.2.
From the above analysis, we show that the efficacy of our framework lies in incorporating the virtual testing performance to provide the feedback (meta gradients) to optimize the training process, which enables the model to learn how to train itself to exploit the desired general knowledge for generalization to unseen concepts, such novel privacy attributes and novel attack models. Note that we do not aim to use the query set to simulate the real testing scenario (which is unknown during training) for model to learn. Instead, we only need the query set to contain novel concepts w.r.t. the support set, so as to provide an effective feedback to drive the model's learning to learn more set-generalizable knowledge and less set-specific knowledge.
4.2 Set Construction
Based on the above virtual training and testing scheme, we can achieve the two target objectives: generalizing to novel privacy attributes and novel attack models, in a unified framework by construct- ing respective support and query sets as follows: Given the training data Xtvain and set of privacy attack models (for training) f we construct a support set D. and query set D,, at the beginning of each training iteration. Specifically, for every odd-numbered iteration, D, and D, are constructed by splitting the training data Xtrain to construct two subsets: X, and Xg. where X, contains novel privacy attributes w.r.t X, (i.c., D.- X, and D, X, in this case). Thus at each odd-numbered iteration, we train the anonymization model to learn to generalize to novel privacy attributes via the virtual training and testing scheme based on the constructed X, and X2. For every even- numbeed iteration, D. and D., are constructed by splitting the privacy attack models ffrain to construct Sf" and 1e" that contain different attack models (i.e., D. J and D. = Sg in this case). Thus at each even-numbered itcration, we train the anonymization model to learn to generalize to novel attack models based on f!" and f." In this odd-even way, we alternate the learning of attribute- wise and mosel-wise generalization iteratively, thus enhancing both generalization capabilities of the model. This means that, due to the unified nature of our scheme, enhancing the model's both generalization abilities has now been reduced to a straightforward designing of their respective set construction methods. Below, we separately discuss each construction method.
A. Constructing sets w.r.t. novel privacy attributes. With respect to the attribute labels, we construct the support set and query set in two steps. Step (A1) At the beginning of each epoch, we intentionally split the training data Xtvain into two subsets (X1, X2). where X2 contains data with novel privacy attributes w.r.t. X,. Step (A2) At the start of every odd odd-nembered training iteration,

---

we select a batch of data from the first subset X1 to construct support set X,, and select batch of data from the second subset X₂ to construct query set Xg.
B. Constructing sets w.r.t. novel attack models. In a similar manner, we also construct the support and query sets w.r.t. privacy attack models in the following two steps. Step (B1) At the start of each epoch, we split all models in ftrain into two subsets [f with different privacy attack models. Step (B2) Then at the start of every even-numbered training iteration, we randomly select one model from f as the support set f." and one model from f/²" as the query set fg"
Note that at each odd ond-numbered iteration (for attribute-wiss generalization learning), randomly select one privacy attack model from firain for anonymization model's training (all models in fhrgin are trained by the adversarial learning algorithm). At each even-numbered iteration (for model-wise generalization learning), we randomly sample a batch of data from Xtrain to train the anonymization model. Moreover, to help the anonymization model to cover wide range of possible attribute shift (and attack model shift) from the provided data set (and model set), instead of fixing (X1, X2) (and {ff f/") }) during the whole training process, at the beginning of each training epoch, we re-split the training data Xtrain (and training attack models firain) to reconstruct X2] (and {f f2 }). It is worth mentioning that our method is convenient to use, since we only need to construct the corresponding sets via the above strategy and change the model training loss without the need to change model structures.
Notewise, an alternative approach to achieve both generalization objectives in a unified framework is to perform both atribute-wise and model-wise learning simultaneously in each iteration, which is also effective for enhancing model generalization capability w.r.t. the above-mentioned aspects, while our method with odd-even alternation performs slightly better in both training convergence and performance (see Sec. 5.5), since it decouples the twofold generalization problem into two individual ones solve, and thus eases the model training and obtains better performance.
4.3 Overall Training and Testing
During training, we follow previous works [13, 14] to alternatively update the three models (fP, f and (2) by using the adversarial training algorithm. At the start of each iteration for updating the anonymization model fD, we first construct the support set and query set w.r.t. privacy altributes in odd-numbered iterations and privacy atlack models in evenenumbered iterations following Sec. 4.2. Note that, here we only focus on the iterations of updating fD, i.e.. we only count the number as odd or even for iterations of updating f After that, the constructed support set and query set are used to train the anonymization model o through the virtual training and testing scheme as discussed in Sec. 4. Hence, we alternatingly deal with the attribute-wise and model-wise generalization problem over iterations, achieving the tackling of both problems. The training for updating f and jP remains the same as previous works [13, 14], detailed in Supplementary. During testing, we follow the evaluation procedures of [13, 14, 28] to evaluate the model's 's generalization ability to novel privacy attack models and novel privacy attributes. We additionally provide an algorithm about the overall training scheme in Supplementary.
5 Experiments
Previous PPAR works have proposed three evaluation protocols: evaluation on novel privacy attributes only [28]; evaluation on novel attack models only [13, 14]; evaluation on known privacy attributes and attack models [28, 15], and each protocol has its own experimental setting. Following these works, we also evaluate our framework under these three protocols. Besides, to evaluate model generalization capability more comprehensively, we further introduce new evaluation protocol: evaluation on both novel privacy attributes and novel privacy attack models together. In each of the four protocols: we are given the training data Xtrain and training (attack) models firai for model training; after training,
evaluate the trained anonymization model on the testing data Xtest with testing (attack) models fiest- The difference between these four protocols lies in whether Xirain and Xtest (or fifain and flest) contain the same privacy attributes (or attack models).
Benchmarks. Following previous works [13, 14, 28], we conduct experiments using two benchmarks. The first benchmark. HMDB51 11-1-VVIS VISPR, is comprised of HMDB51 [31] dataset and VISPR [30] dataset. HMDB51 [31] is a collection of videos from movies and the Internet. has 6,849 videos with 51 action categories. VISPR [30] is collection with a diverse set of personal privacy information. Following [13, 14, 28), we use PA-HMDB [13] that contains 515 video samples with both human
action and privacy attribute labels to serve as the testing set of HMDB51-VISPP The second
7

---

Algorithm 1: Overall Training Scheme
Given Xtrain and / training; Xtevt fi testing. Initialize @.
E epocks
Process Xtrain and F Ferain following Step (Al) Step (B1) Sec. respectively.
T' O.
sterations adversarial training do if I is updating anonymization model f then I is odd then
X. to X. to from (enins following Step (A2)
eise
I1
Construct Je and I from frain lollowing Step (B32) in Sec. 4.2.
end
Calculate the virtual training loss Lu D. fie., X fe ( using Eq 4: Ly (n)) L(0)
Calculate updated version anonymization model (0' using Eq. 5: o -V
Calculare the virtual testing loss Les on De (i.., Nuee or fo using Eq. 6. Lageldd) )= Lid', Duge).
Update using Eq.
9- v (o) (6) + Logee ( a (o
I= T else if is updating recognition model then Randomly sample batch of data from X ceoin updale f using Ey. 2.
20
else
21
Randomly sample batch from Nerain to update f using Eq.
22
end
23
end
24
end
benchmark, UCF101- -ISPPR, consists of UCF101 [29] dataset and VISPR [30] dataset. UCF101 [29] has 13,320 videos taken from 101 action categories. Following [13, 14, 28), we use the testing sets of
UCF101 [29] and VISPR [30] for evaluation in this benchmark. all protocols, Xtrain (or Xtest) ) is
created by sampling from the training set (or the testing set) of the benchmark.
Evaluation Method. In all protocols, follow previous works [13, 14] to evaluate methods in two folds: (1) action recognition performance whether the anonymous videos still contain the action clues to enable the action recognition model to maintain satisfactory performance; (2) privacy preservation performance whether the anonymous video can lead to poor performance of the privacy attack models. As for the evaluation of action recognition, we directly apply the trained f and f to the testing data set Xtest (i.e.. [ā (Xtest))) and compute the action classification accuracy: the higher the better. As for the evaluation of privacy preservation, to empirically verify that fo prohibits reliable privacy prediction (attack) for each model in the testing model set fir fiest we conduct the evaluation as follows: before evaluation, we train each model in flest for privacy attribute classification; during evaluation, apply each trained model in fees to perform privacy attribute classification on anonymous videos (X (anonymized by (D)) and compute the classification accuracy. The highest accuracy achieved among all models of fe will be by default used to indicate the privacy leakage risk of fD, where lower value denotes better privacy preservation.
Raw
Fhone
Fiotre
Figure The frames before and after our anonymization. More re- sults are in Supplementary.
5.1 Protocol A: Evaluation on Both Novel Privacy Attributes and Novel Attack Models
Experimental Setting. In this evaluation protocol, we follow previous work [28] to set up the training data Xtrain and testing data Xtest: making X Xtrain have no
Table Results on generalization to both novel privacy attributes and attack models.
HMDBSL-VISED VISPR
UCFIBL-VISPR
overlapped privacy attributes with Xteat- Besides, we
Methook
Action (Top-1 ")
Prinacy (cMAP J)
Action (Top-1 1)
Privayy (cMAP
follow [13, 14] to set up the training models firai
and the testing models fPr with totally different pri-
Raw Dsta
68.5
66.3
67.4
42.4
Downsmmple (28
49,0
63.1
50.2
vacy attack models. This means in this protocol, we need simultaneously handle novel privacy attributes and novel privacy attack models. The detailed list of training/testing attributes and attack models can be found in Supplementary.
Downsmmpl (28)
$2.5
43.3
Ohf Biackening [28]
40.4
61
08.3
Ohi-S SinnnSSnn [28]
41.7
69.2
Ohi-WasBBlrr (18)
60.8
49.5
VIIA [13]
67.9
54.5
66.5
30.2
SPhci (25)
67.5
51.8
65.7
293

---

Implementation Details. Following [13, 14], we use the Image Transformation model [37] as fd, the action recognition model C3D [32] as f and set of privacy classification models from MobileNet- V2 [33] family as ffrain- Since we use C3D [32], we need to split the videos into clips with a fixed frame number. We use the clip with of 16 frames with skip rate of The spatial resolution of each video is resized into 112 x 112. For a fair comparison, our method and compared methods use the same model architectures. On each benchmark, we construct the support set with the videos containing 60% of the privacy attributes in the training data X Xtrain and use the remaining training data to construct the query set. We set y (in Eq. 1) as 0.4, the learning rate a for virtual training (in Eq. 5) as 5e 4, and the learning rate 3 for meta-optimizztion (in Eq. 8) as le 4.
Experimental Results. As shown in Tab. 1, after applying our virtual training and testing scheme to the basic model, our framework, i.e., Ours (full), improves the model performance by a large margin, which demonstrates the effectiveness of our framework attributes and novel attack models simultaneously. "Raw Data" denotes directly using raw (clean) videos for testing. Moreover, compared to existing methods, Ours (full) achieves significantly better privacy performance while keeping the best action performance, demonstrating the superiority of our framework. Qualitative results are shown in Fig. 2.
generalizing to novel privacy
5.2 Protocol B: Evaluation on Novel Privacy Attributes Only
Experiment Setting, In this experiment, we follow the evaluation protocol in [28], which focuses on the generalization to novel privacy attributes, to set up Xtrain and test containing different attributes, and use the same privacy classification model for both training and testing.
Table 2: Results on generalization to novel privacy attributes.
HMDB51 VISPR
UCP101 VISPR
Methods
Action (Top-1
Privacy (cMAP
Action
Privacy (cMAP))
(Top-1 1)
Raw Data
47.8
61.7
62.9
58.3
Downssmple 2s (28
18.5
58.8
54.1
52.2
Dommmample 4x [28
32.4
55.3
39.7
41.5
Implementation Details. Following [28], we utilize UNet [34] as fd, R3D-18 [36] as fa, and ResNet-50 [35] as f Other implementation details remain the same as Protocol A.
Olf- Black-niniing [26]
20:7
57.0
53.1
$3.6
Dikt-SinnSälur [26]
21.3
56.9
55.6
$3.7
Oid--Wekkklrr [18]
23.5
57.3
61.5
55.8
VITA (13)
45.1
53.4
02.1
49.6
SPAct (28)
44.7
45.3
62.0
47.1
BDQ (15)
46.6
52.2
62.3
48.8
Basic Medel
44.5
54.0
61.8
50.1
For a fair comparison, our method removes the virtual training and testing iter- ations w.r.t. model-wise generalization and only carries out the attribute- wise iterations, which is denoted as Ours (only attribute) in Tab. 2. Compared to other methods, Ours (only attribute) brings obvious performance improvement, demonstrating our method's ' effectiveness in tackling privacy- attribute generalization problem individually. Ours (full) denotes the model trained under Protocol A, and here we also directly evaluate its performance under the current protocol. As shown, this model also achieves remarkable trade-offs, demonstrating our method trained for both generalizations can also well handle the generalization scenario where only the attributes are novel.
Experimental Results.
Ours (aal)
Oun (only anmibute)
47.2
47.5
43.2
41.5
62.5
62.6
32.0
32.2
5.3 Protocol C: Evaluation on Novel Privacy Attack Models Only
Experiment Setting. We follow the protocol used in [13, 14), which focuses on the generalization to novel privacy attack models, to set up the training and testing models frain and first containing totally different privacy attack models. The training data X and testing data Xtest contain the same privacy attributes.
Table 3: Results on generalization to novel privacy attack models.
HMDBSI-VISPR
UCFIOL-VISPR
Methods
Action 1)
Privacy (cMAP
Action
Privacy (cMAP
(Top 11)
Raw Duta
67.9
67.4
46,3
Downeomple 2x (28)
53.9
23.2
50.2
38.3
Dovmoomple 4x [28]
$2.6
72.9
43.3
34.7
Implementation Details. For fair comparison, we use the same model architectures as [13, 14] (Sec. 5.1). Other implementation details are same Protocol A.
Ohh--Baackeing [18)
40.2
48.5
31.1
Dibl-SmmngRlir [25]
41.8
71.6
49.2
30.8
ChhtWcckBBar (28)
43.3
71.7
49.5
VITA [13)
66.6
64.8
56.5
34,2
SPAce [25)
65.3
66.7
65.7
35.0
Model
56.2
Experimental Results. For fair comparison, in this experiment, our method only adopts virtual training and testing iterations w.r.t. model-wise generalization, which is denoted as Ours (only model). As shown in Tab. 3, compared to existing methods and the basic model, Ours (only model) can reduce the privacy leakage by large margin, showing our method's 's effectiveness in misel-wise generalization. Furthermore, Ours (full) can also outperform existing methods on the privacy performance while maintaining the best action performance, demonstrating our framework trained for both generalizations can also well handle the generalization where only the attack models are novel.
Ours (oniy model)
Oum (fall)
67.0

---

5.4 Protocol D: Evaluation on Known Privacy Attributes and Attack Models
Experiment Setting. We follow the protocol used in [28], which also proposes to conduct evaluation on known privacy attributes and privacy attack models, to use Xtrain and Xtest that contain the same attributes and adopt the same privacy classification model for both training and testing.
Table 4: Results on generalization to known
privacy attributes and privacy attack models.
HMIDSSIVISPP
UCFIOL-VISPR
Methods
(Top-1 ")
(cMAP
(Tap-1 T) (cMAP
20.6
Downsmmple 2x [28
36.1
61.2
54.1
57.2
Dovmammple 4x [2X
25.8
41.4
39.7
50.1
Implementation Details. For a fair comparison, we follow the previous work [28] using the same model
Olt- Blackennng [28]
24.2
63.88
53.1
36.4
thhi-Simngfller (26]
16.4
64.4
55.6
55.0
(iki-Waikklrr (38]
41.7
60.4
61.5
63.5
architectures (see Sec. 5.2). Other implementation
SPAut (28]
VITA (13]
43.1
42.3
62.3
62.7
62.1
62.0
574
55.3
details are the same as in Protocol A.
BDQ (15)
43.3
61.8
62.3
53.5
Basic Model
42.1
62.8
62.0
Experimental Results. In this protocol, since we only have one privacy classification model, we cannot perform the virtual training and testing iterations w.r.t. model-wise generalization, but we can still carry out attribute-wise iterations, as Xtrain contains multiple privacy attributes. Therefore, in this protocol, we adopt Ours (only attribute) to fairly compare with others. As shown Tab. 4, the downsampling-
Our (fail)
Oun (only anribute)
43.4
43.5
61.0
61.2
62.5
62.6
53.2
baasd based methods bring obvious performance drop in action recognition. In contrast, our method achieves the closest action recognition performance to Raw Data, and meanwhile outperforms other learning-basee training the model to capture more general knowledge, our framework can help the model to better understand the given video and find its privacy information, which thus can generally improve model performance. We can also see that Ours (full) shows comparable results with Ours (only attribute).
methods [13, 28, 15] in privacy preservation. This can be attributed to that by
5.5 Ablation Studies
Impact of alternate learning with odd and even num- bered iterations. To enhance model generalization abil- ities to both novel privacy attributes and novel privacy attack models, we perform the attribute-wise virtual train- ing and testing and model-wise virtual training and testing for odd-numble add-numbered iterations and evenenemeered iterations, respectively. In this alternate way. the model can learn to improve both abilities in the whole training process, si- multaneously. To investigate the impact of alternate learn- ing, we evaluate the variant by using both virtual training and testing strategies at each single iteration, denoted as mtri+ learning. As shown in Tab. 5 and Fig. 3, al- ternate learning outperforms the variant and shows faster convergence, which demonstrates its effectiveness. More ablation studies are in Supplementary
Table 5: Ablation results of model+ learning and alternate learning.
Metho's
HMDEST VISPS
UCF101 VISeR
Aztion f
Paiaay
Action
Priwxy
striemmdel loening
579
47.9
66.6
22,4
aliemane leaniing
68.1
47.4
67.0
21,8
attrl leaming
altemate
Loss
3000
6000
9000
Iterations
Figure The loss curves of modrl learning and alternate learning.
Conclusion
In this paper, we have proposed a unified framework that improves the generalization ability of anonymization model, through simultaneously improving its performance for handling novel privacy attributes and novel privacy attack models. By carefully constructing the support set and query set with different attributes and attack models, our framework trains the anonymization model with virtual training and testing scheme and guides it to learn knowledge that is general to scenarios where the privacy attributes and attack models are novel. Extensive experiments have shown the efficacy of our framework under different settings.
7
Acknowledgments
This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA 1004100) 100, National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: S0-G--PPDD-222-1-02222 Singapore Ministry of Education (MOE)
10

---

AcRF Tier under Grant MOE-T2EP20222-0002 National Research Foundation Singapore through AI Singapore Programme under Grant AISG- 12211123111 and SUTD SKI Project under Grant SKI 2021_02_06
References
[1] Marco Buzzelli, Alessio Alb e, and Gianluigi Ciocea. A vision-based system for monitoring elderly people at home. In Applied Sciences, 10(74;374, 2020.
[2] Forougii Homa, Aski Baharak Shakeri, and Pourreza Hamidreza. Intelligent video surveillance for monitoring fall detection of elderly in home environments. In International Conference on Computer and Information Technology, IEEE, 2008.
[3] Francesco Pittaluga, Sanjeev Koppal, and Ayan Chakrabarti. Learning privacy preserving encodings through adversarial training. In Winter Conference Applications of Computer Vision (WACV), pages 791-799, IEEE, 2019.
[4] Taihong Xiao, Yi-Hsuan Tsai, Kihyuk Sohn, Manmohan Chandraker, and Ming-Hsuan Yang. Adversarial learning of privacy-preserving and task-oriented representations. In AAAI Confer- ence on Artificial Intelligence, volume 34, pages 12434-12441, 2020.
[5] Rajput, Amitesh Singh, Balasubramanian Raman, and Javed Imran Privacy-preserving human action recognition as a remote cloud service using RGB-D sensors and deep CNN In Expert Systems with Applications, 152(2020): 113349, 2020.
[6] Jihun Hamm. Minimax filter: learning to preserve privacy from inference attacks. In The Journal of Machine Learning Research, 18(1):4704 4734, 2017.
[7] Graham Cormode. Individual privacy population privacy: Learning to attack amonymization.
In arXiv preprini arXiv: 1011, 2010.
[8] Ji Dai, Behrouz. Saghafi, Jonathan Wu, Janusz Konrad, and Prakash Ishwar. Towards privacy- preserving recognition of human activities. International Conference on Image Processing
(ICIP), pages 4238 4242, IEEE, 2015.
[9] Michael Ryoo, Kim Kiyoon, and Hyun Yang. Extreme low resolution activity recognition with multi-siamese embedding learning. In AAAI Conference on Artificial Intelligence, 2018.
[10] Michael S Ryoo, Brandon Rothrock, Charles Fleming, and Hyun Jong Yang. human activity recognition from extreme low resolution. In AAAI Conference on Artificial Intelligence, 2017
[11] Zhongzheng Ren, Yong Jae Lee, and Michael S Ryoo. Learning anonymize faces for privacy preserving action detection. In European Conference on Computer Vision (ECCV), pages 620-636, 2018.
[12] Zhixiang Zhang, Thomas Cilloni, Charles Walter, and Charles Fleming. Multi-scale, class- generic, privacy-preserving video. In Electronics, 10(10): 1172, 2021.
[13] Zhenyu Wu, Haotao Wang, Zhaowen Wang, Hailin Jin, and Zhangyang Wang. Privacy- preserving deep action recognition: An adversarial learning framework and a new dataset. In Transactions on Pattern Analysis and Machine Intelligence (TPAMI). IEEE, 2020.
[14] Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. Towards visual recognition via adversarial training: A pilot study. In European Conference Computer Vision (ECCV), pages 606-624, 2018.
[15) Sudhakar Kumawat and Hajime Nagahara PrivacyPrreeering Action Recognition via Motion Difference Quantization. European Conference on Computer Vision (ECCV), pages 518-534. 2022.
[16] Guoqiang Peter Zhang. Neural networks for classification: survey, In Transactions on Systems, Man, and Cybernetics, pages 451-462, IEEE, 2000.
[17] Mireshghallah Fatemehsadat, Taram Mohammadkazem, Vepakomma Praneeth, Singh Abhishek, Raskar Ramesh, and Esmaeilzadeh Hadi Privacy in Deep Learning: Survey. In arXiy preprint arXiv:2004.1225 2020.

---

[18] Dima Damen, Hazel Doughty, Farinella Giovanni Maria, Sanja Fidler. Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and others Scaling egocentrie vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720-736, 2018.
[19] Zehua Sun, Qiuhong Ke, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, and Jun Liu Human action recognition from various data modalities: A review. In IEEE transactions on pattern analysis and machine intelligence, 2022.
[20] Santosh Kumar Yadav, Kamlesh Tiwari, Hari Mohan Pandey, and Shaik Ali Akbar A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions. In Knowledge-Bacd
Systems, 2021.
[21] Chelsea Finn, Abbeel Pieter, and Levine Sergey. Model-agnostic meta-learning for fast adap- tation of deep networks. In International Conference on Machine Learning (ICML), pages 1 1126-1135, PMLR, 2017.
[22] Alex Nichol, Achiam Joshua, and Schulman John. On first-order meta-learning algorithms. In arXiv preprint 1803.0999,, 2018
[23] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Neural Information Processing Systems (NeurIPS), 2019.
[24] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Neural Information Processing Systems (NeurIPS), 2017.
[25] Mei Wang, Yaobin Zhang, and Weihong Deng. Meta Balanced Network for Fair Face Recogni- tion. In Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pages 8433-8448,
IEEE, 2022.
[26] Yu- Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-Learnin to Detect Rare Objects. In International Conference on Computer Vision (ICCV), IEEE, 2019.
[27] Aniruddh Raghu, Jonathan Lorraine, Simon Kornblith, Matthew McDermott and David K Du- venaud. Meta-learning to improve pre-training. In Advances in Neural Information Processing Systems (NeurlPS), pages 23231-23244, 2021.
[28] Ishan Rajendrakumar Dave, Chen Chen and Shah Mubarak. Spact: Self-supervieed privacy preservation for action recognition. In Computer Vision and Pattern Recognition (CVPR), pages 20164-20173 IEEE/CVE, 2022.
[29] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv:1212.04000 2012.
[30] Tribhuvanesh Orekondy, Bernt Schicle, and Mario Fritz. Towards visual privacy advisor: Understanding and predicting privacy risks in images. International Conference on Computer Vision (ICCV), 2017.
[31] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. International Conference on Computer Vision (ICCV), 2011.
[32] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In International Conference on Computer Vision (ICCV), pages 4489-4497, 2015.
[33] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In Computer Vision and Pattern Recognition (CVPR), pages
4510-4520, 2018.
[34] Olaf Ronneberger, Philipp biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234-241, 2015.
Fischer, and Thomas Brox. Unet: Convolutional networks for
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[36] K. Hara, H. Kataoka, and Y. Satoh. Towards good practice for action recognition with spa- tiotemporal 3d convolutions. In International Conference on Pattern Recognition (ICPR), pages 25 2516-2521, 2018
[37] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision (ECCV), 2016.
12

---

[38] Shi, Y., Seely, J., Torr, P. N, S., Hannun, A., Usunier, N., and Synnaeve, G. Gradient matching for domain generalization. In International Conference on Learning Representations (ICLR), 2022.
[39] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll' ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[40] Joseph Redmon, Santosh Divvala, Ross Girshick. and Ali Farhadi. You only look once: Unified, real-time object detection. In Computer Vision and Pattern Recognition (CVPR), pages 779-788, 2016.
[41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision (ECCV), pages 630-645, 2016.
[42] Christian Szegedy. Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions In Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
[43] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision In Computer Vision and Pattern Recognition (CVPR), pages 2818-2824 2016.
[44] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In arXiv preprint 1rXiv:1704.04886 2017.
13


Reviews:
[{"review_id": "UEBy5rqDl8", "reviewer": "Reviewer_hKip", "summary": "To deal with novel privacy attributes and novel privacy attack models that are unavailable during the training phase, this paper applies the meta learning to the privacy-preserving action recognition. As a result, this work improves both generalization abilities above (i.e., generalize to novel privacy attributes and novel privacy attack models) in a unified manner.", "questions": "See above.", "limitations": "None.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper is well-written, well-organized and easy to understand. The arguments and conclusions are clear and explicit.\n2.\tThis paper proposes a solution to address a important and interesting problem, i.e., preventing privacy leakage while maintaining action clues for activation recognition.\n", "weaknesses": "1.\tThis paper claims that the proposed method is driven to learn more generalizable knowledge and can help remove potentially unseen privacy attributes (and defend against unknown privacy attackers). Although the authors provide some theoretical understanding about the objective function in the appendix, are there any theoretical guarantees about the generalization of the proposed method?\n2.\tThis paper claims that guide anonymization models to learn such generalizable knowledge is challenging, but is seems like can be directly solved by employing the meta learning. This makes this paper appear to utilize meta-learning for action recognition, which limits the novelty. What are the challenges in applying meta-learning?\n3.\tMore comparison methods from the last two years need to be compared to verify the effectiveness and generalization of the proposed method.\n4.\tIt would have been better if the authors could provide the standard deviation of the experimental results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1:** *\"Are there any theoretical guarantees about the generalization of the proposed method?\"*\n\n**A1:** Given the training set $D_{train}$, the conventional training scheme, which directly training the model $\\phi$ on the whole training set $D_{train}$, can be formulated as: \n$$\\min_{\\phi} L(\\phi, D_{train}). \\tag{1}$$\n\nAs mentioned in the main paper, we split the training set into two sub-sets, i.e., $D_{train} = \\lbrace D_s, D_q \\rbrace$, where $D_s$ and $D_q$ are the support set and the query set, respectively.\nIn our appendix, we have theoretically demonstrated that our framework's training loss can be reformulated as:\n$$\\min_{\\phi} L(\\phi, D_s) + L(\\phi, D_q)- \\alpha \\big{(} \\nabla_\\phi L(\\phi, D_s) \\cdot\\nabla_\\phi L(\\phi, D_q)\\big{)}. \\tag{2}$$\n\nGiven the training set $D_{train} = \\lbrace D_s, D_q \\rbrace$, the first two terms of Eq. 2, i.e., $\\min_{\\phi}\\;L(\\phi, D_s) + L(\\phi, D_q)$, which denote the training on both $D_s$ and $D_q$, is analogous to the training on the combined training set (whole training set) $D_{train}$, which can be formulated as:\n$$\\min_{\\phi} L(\\phi, D_s) + L(\\phi, D_q) \\approx \\min_{\\phi} L(\\phi, D_{train}). \\tag{3}$$\n\nThen, we substitute Eq. 3 into Eq. 2, reformulating our framework's training loss as:\n$$\\min_{\\phi} L(\\phi, D_{train})- \\alpha \\big{(} \\nabla_\\phi L(\\phi, D_s) \\cdot\\nabla_\\phi L(\\phi, D_q)\\big{)}. \\tag{4}$$\n\nComparing the conventional training loss (Eq. 1) and our training objective (Eq. 3), we can observe that our framework introduces an additional term: i.e., $\\min_{\\phi}  - \\alpha \\big{(} \\nabla_\\phi L(\\phi, D_s) \\cdot\\nabla_\\phi L(\\phi, D_q)\\big{)}$. In our appendix, we have mathematically demonstrated that this additional term benefits from the design of our meta-framework and is able to learn more general gradients that avoid overfitting to some specific training tasks. Thus, this additional term drives and guarantees the our framework to learn more generalizable knowledge compared to the conventional training method.\n\nIn summary, by theoretically deducing and comparing the training objectives between the conventional training and our approach, it shows that our approach enables more generalizable learning.\nFurthermore, the results of the generalization experiments presented in Table 1, 2, and 3 (in our main paper) clearly indicate that our method consistently outperforms other methods by a significant margin, demonstraing our method's effectiveness on generalization. (The above theoretical deduction will be included into our main paper.)\n\n>**Q2:** *\"What are the challenges in applying meta-learning?\"*\n\n**A2:** Privous works [13, 14, 25] have shown that the generalization problem in privacy preserved action recognition is important and challenging. Facing this challenging task, although we propose a novel Meta-learning framework to address this issue, it is not straightforward to simply apply meta-learning.\n\nMeta-learning is primarily introduced to tackle the few-shot learning problem [a, b]. However, in this paper, we aim to address the privacy generalization problem, where the direct application of traditional meta-learning is unsuitable. To tackle this challenge, we carefully devise a Set Construction method to privide effective support and query sets by intentionally partitioning these two sets to contain completely different attributes (and attack models), which build the basis for our meta-framework.\n\nAs we propose to handle both attribute-wise generalization and attack-model-wise generalization, it is still difficult to solve this problem with meta-learning. Therefore, we further propose an odd-even alternation learning strategy (see line 276 of our main paper), which decouples the twofold generalization problem into two individual ones to solve, thus enabling meta-learning in a unified framework for both generalization purposes.\n\nIn this paper, to address the privacy generalization problem, we propose a novel meta-framework with the set-construction design, which is different from traditional few-shot-based meta-learning. To analyze such newly-designed framework, we also provide a theoretical deduction that showcases the efficacy of our apporach, which facilitates the understanding, utilization, and extension of our approach by future research.\n\n[a] Finn C, et al. Model-agnostic meta-learning for fast adaptation of deep networks, ICML, 2017.\n\n[b] Nichol A, et al. On first-order meta-learning algorithms, arXiv, 2018.\n\n>**Q3:** *\"More comparison methods from the last two years need to be compared.\"\n\n**A3:** Thank you for suggestion. To the best of our knowledge, in our paper, we have comapred our method with all methods published in the last two years, including [13] (TPAMI 2020), [25] (CVPR 2022), [15] (ECCV 2022).\n\nFor a more comprehensive comparison, here we compare our method with STPrivacy [c], which is the only arXiv paper before our submission. We conduct experiments on HMDB51-VISPR using the same backbone as STPrivacy for a fair comparison. The results below demonstrate that our approach can outperform STPrivacy.\n\n| Method | Action $\\uparrow$ | Privacy $\\downarrow$ |\n|---|---|---|\n| STPrivacy [c] | 50.7 | 72.5 |\n| Ours  | **51.2** | **70.9** |\n\n[c] Li M, et al. STPrivacy: Spatio-Temporal Tubelet Sparsification and Anonymization for Privacy-preserving Action Recognition. arXiv, 2023.\n\n>**Q4:** *\"Provide the standard deviation of the experimental results.\"\n\n**A4:** Thanks for valuable comments. The result we reported in the paper is the average of five repetitions. Following your suggestion, we will add the deviation after each reported result. Here, we show the resluts with deviation on HMDB51-VISPR under Protocal A as follows:\n\n| Method | Action $\\uparrow$ | Privacy $\\downarrow$ |\n|---|---|---|\n| Basic Model | 67.3 ($\\pm$ 0.3) | 55.2 ($\\pm$ 0.6) |\n| Ours  | **68.1** ($\\pm$ 0.2) | **47.4** ($\\pm$ 0.4) |\n\nWe will add the deviation results for all Protocals and datasets into our main paper.\n"}, {"review_id": "2oHPEWvzW8", "reviewer": "Reviewer_4pCF", "summary": "Considering that current privacy-preserving action recognition are difficult to cope with novel privacy attribute and privacy attack model, this paper propose a Meta Privacy-Preserving Action Recognition (MPPAR) framework to improve generalization abilities. Specifically,  inspred by meta learning, the authors construct support set for virtual training, and query set for virtual testing. Then authors design virtual training and testing scheme to drive the model learn more generalizable knowledge that can improve the model generalization capability.", "questions": "Seen Weakness", "limitations": "Authors should verify the proposed method in more complex action data.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Based on meta-learning, the authors design virtual training and virtual testing processes, thereby improving the generalization of privacy-preserving action recognition.\n2. The proposed method effectively defends against the privacy attack model on the basis of maintaining the accuracy of action recognition.", "weaknesses": "1.  The description of the task is not clear, making it difficult to understand the mission objectives. To protect the privacy of individuals in the video and identify the action behavior, why not directly convert the RGB video into a skeleton sequence？It is well known that actions can already be well identified by skeletal sequences alone, and other information other than the actions of the actor can be removed from the video.\n2. The description of the experimental setup is not clear enough. For the Privacy-Preserving Action Recognition task, authors  use the data that contains 7 privacy attributes (i.e., color, gender, complete face, partial face, semi-nudity, personal relationships, social relationships) for training, and use the data that contains other 7 privacy attributes (i.e., hair color, race, sports, age, weight, landmark, tattoo) for testing. Why did authors choose these attributes for training and testing?\n3.  The authors used the C3D model for action recognition, but the recognition accuracy of the original videos in Table 1、2 and 3, was inconsistent. The original video should be unencrypted data, why they have different recognition accuracy under the same recognition model？\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1:** *\"The description of the task is not clear, making it difficult to understand the mission objectives. To protect the privacy of individuals in the video and identify the action behavior, why not directly convert the RGB video into a skeleton sequence? It is well known that actions can already be well identified by skeletal sequences alone, and other information other than the actions of the actor can be removed from the video.\"*\n\n**A1:** \nWhile skeleton-based methods could handle some level of privacy concerns, the research regarding privacy preservation in RGB-based action recognition still remains important and practical. Here are the reasons:\n\n1. In many scenarios, besides action information, it is sometimes also necessary to incorporate information beyond action into the action recognition model. For instance, in the kitchen scenario [a] which contains many fine-grained actions, there are many actions that can be similar, such as wash dishes and cut vegetables (both require one hand to hold onto the object while the other hand performs periodic shaking motions). In this case, accurately recognizing the action requires richer contextual information from the background and surroundings. In such a scenario, simply using the skeleton to remove all other information will lead to performance degradation (as discussed in paper [b]). \nThis indicates that in many scenarios, we still require RGB videos to provide reliable action recognition.\nTherefore, In RGB-based action recogtion, learning how to remove privacy information while retaining both action information and related visual cues for accurate action recognition becomes extremely important.\nRecently, it has attracted increasing attention and a series of works including TPAMI2020 [13], EECV2022 [15] and CVPR2022 [25] has been proposed, reflecting the significance of tackling privacy concerns within RGB-based action recognition.\n\n2. Furthermore, RGB-based action recognition has already been extensively deployed in many household applications [c], such as Amazon Echo and Google Nest Cam. Therefore, the privacy preservation for RGB-based action recognition is of utmost urgency. \n\nWe will include the above analysis into our main paper to motivate the task clearly.\n\n[a] Damen D, et al. Scaling egocentric vision: The epic-kitchens dataset, ECCV, 2018.\n\n[b] Sun Z, et al. Human Action Recognition from Various Data Modalities: A Review, TPAMI, 2023.\n\n[c] Yadav S K, et al. A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions, Knowledge-Based Systems, 2021.\n\n>**Q2:** *\"The description of the experimental setup is not clear enough. For the Privacy-Preserving Action Recognition task, authors use the data that contains 7 privacy attributes (i.e., color, gender, complete face, partial face, semi-nudity, personal relationships, social relationships) for training, and use the data that contains other 7 privacy attributes (i.e., hair color, race, sports, age, weight, landmark, tattoo) for testing. Why did authors choose these attributes for training and testing?\"*\n\n\n**A2:** \nIn the previous attribute-wise generalization protocal [25], it has bulit a standard evaluation setting with 7 privacy attributes (i.e., color, gender, complete face, partial face, semi-nudity, personal relationships, social relationships) for training and  other 7 privacy attributes (i.e., hair color, race, sports, age, weight, landmark, tattoo) for testing. Therefore, in this work, for fair comparison, we follow this evaluation setting. We will clearly introduce this part in our main paper.\nBesides, we also randomly split the training/testing attributes for 3 times, obtaining 3 evaluation settings (namly, $S_1$, $S_2$, and $S_3$). We evaluate our method on HMDB51-VISPR under the 3 evaluation settings. The results below show that our method achieves the significant improvements on all evaluation settings.\n\n\n| Method  | $S_1$ | $S_2$ | $S_3$ | Action $\\uparrow$  | Privacy $\\downarrow$ | \n|---|---|---|---|---|---|\n| Basic Model | $\\surd$  |   |   | 43.7 | 53.4 |\n| Ours | $\\surd$  |   |   | **45.2** | **46.9** | \n| Basic Model |     | $\\surd$ |  | 45.2 | 57.2 |\n| Ours |   |  $\\surd$  |  | **45.9** | **48.8** | \n| Basic Model |   |   |  $\\surd$ | 42.6 | 51.9 |\n| Ours |   |  |  $\\surd$ | **43.3** | **47.0** | \n\n\n>**Q3:** *\"The authors used the C3D model for action recognition, but the recognition accuracy of the original videos in Table 1, 2 and 3, was inconsistent. The original video should be unencrypted data, why they have different recognition accuracy under the same recognition model?\"*\n\n**A3:** \nIn PPAR, there are four Protocols which focus on different generalization problems: Protocal A, generalization to novel privacy attributes and novel attack models; Protocal B, generalization to novel privacy attributes only; Protocal C, generalization to novel attack models only; Protocal D, seen privacy attributes and attack models. \nFor a fair comparison, we follow the corresponding standard setting for each protocol. Since different protocals adopt different baseline models or different training data, the experimental results in Table 1, 2, 3, 4 (corresponding to four protocals) in our main paper are different."}, {"review_id": "CJe9fvwLiX", "reviewer": "Reviewer_EKAb", "summary": "The paper proposes a novel privacy-preserving action recognition (PPAR) framework built around the MAML meta-learning framework by Finn et al.. The goal of PPAR is to train an anonymization model that is able to anonymize video data so that 1) an action recognition model can still predict the depicted action in the video and 2) the anonymized video does not reveal sensitive information like the visual appearance of a person to the adversary. The approach consists of three steps, virtual training, virtual testing, and meta-optimization. During the virtual training, the model is trained on a support set and then tested on a disjoint query set. The meta-optimization is then performed based on the testing loss. The proposed approach is experimentally evaluated on multiple datasets and compared to various previous defense approaches for PPAR.", "questions": "- Is it possible that the proposed framework leads to the generation of adversarial examples in the intermediate (anonymized) space, i.e., it somehow \"overfits\" the attack models / architectures? If so, it might be possible that other attack choices would still be able to extract sensitive information from the anonymized images if their attack models are learning different features from the anonymized images? For instance, adversarial examples can also be transferrable between different models and architectures (e.g., CNNs), but ViTs might be robust against these adversarial examples. So attack models based on ViTs instead of CNNs might infer more sensitive information.\n- The approach requires training data with labels for the private attributes available. If no such dataset is available, could the approach still be applied?", "limitations": "Technical limitations are not discussed in the paper. I do not expect negative societal impact, so it is ok for me that it is not discussed in the main paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper is generally well-written and mostly easy to understand. Figure 1 helps to understand the overall approach. Particularly the task is introduced in a way that also people with basic machine learning knowledge should be able to understand it.\n- The meta-learning approach adds an interesting new avenue to PPAR research and offers a practical defense against future attack approaches.\n- The evaluation is extensive, and the results are promising. Since I am not an expert in the PPAR domain, I cannot tell if important related work has been ignored. But the paper investigates four different settings, which seems convincing.", "weaknesses": "- The font size in the figures and tables is a bit small. To improve readability, the font size should be increased in these cases. Also, the space after Fig. 1 should be increased. \n- Some parts of section 4 could be more succinct\n- The Metrics, particularly cMAP, should be formally introduced. \n- The paper can be improved by adding a technical limitations section, discussing, e.g., the additional training time, data requirements, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1:** *\"The font size in the figures and tables is a bit small. To improve readability, the font size should be increased in these cases. Also, the space after Fig. 1 should be increased.\"*\n\n**A1:**  Thank you for the feedback. To improve readability, we will increase the font size in the figures and tables. Additionally, we will adjust the space after Fig. 1 to enhance the overall presentation.\n\n>**Q2:** *\"Some parts of section 4 could be more succinct.\"*\n\n**A2:** Thank you for the comment. We will review section 4 and make the necessary adjustments to make it more succinct while still conveying the essential information effectively.\n\n\n> **Q3:** *\"The Metrics, particularly cMAP, should be formally introduced.\"*\n\n**A3:** In the revised version, we will formally introduce the Metrics, especially cMAP, providing a detailed explanation of its calculation. Next is the introduction of cMAP:\n\nTo assess performance of privacy preservation, we follow previous work [13, 15, 25] using Class-based Mean Average Precision (cMAP) as the evaluation metric. The cMAP metric calculates the Average Precision (AP) per class, which represents the area under the Precision-Recall curve for each attribute. Moreover, the overall performance of a method is determined by averaging the AP scores across all attributes. A lower cMAP indicates better privacy preservation.\n\n\n>**Q4:** *\"The paper can be improved by adding a technical limitations section, discussing, e.g., the additional training time, data requirements, etc.\"*\n\n**A4:** Thanks for valuable suggestion. We will add the following limitation discussion of our method in the main paper:\n\nWe evaluate the training time of our framework, and compare it to the training time of the Basic Model that trains the same network in the conventional training manner. Since our proposed framework conducts both virtual training and virtual testing at each iteration, there is an increase in training time compared to conventional training scheme.  The training time for our method is 1.8 days, while the training time for the Basic Model is 1.4 days (both are trained using a single RTX 3090 GPU).  Despite the increase in training time, our approach (1) preserves the model's original structure, resulting in inference time that is almost the same as the Basic Model; (2) achieves nearly 10\\% performance improvement on average when compared to the performance of the Basic Model, which is very significant.\n\n\n>**Q5:** *\"Is it possible that the proposed framework leads to the generation of adversarial examples in the intermediate (anonymized) space, i.e., it somehow \"overfits\" the attack models / architectures? If so, it might be possible that other attack choices would still be able to extract sensitive information from the anonymized images if their attack models are learning different features from the anonymized images? For instance, adversarial examples can also be transferrable between different models and architectures (e.g., CNNs), but ViTs might be robust against these adversarial examples. So attack models based on ViTs instead of CNNs might infer more sensitive information.\"*\n\n\n**A5:** Thanks for your valuable suggestion. To demonstrate the generalization of our method to Vision Transformers (ViTs), we conduct experiments by adopting the widely-used ViT model [a] rather than CNN model.\nThe experiments are conducted on HMDB51-VISPR, where the results are shown below.\nWe can see that when tested on unseen ViT models, our model (trained with CNN models) also shows remarkable generalization performance in privacy preservation, which is competitive to the results obtained when testing on unseen CNN models.\n\n| Method  | Testing on CNN | Testing on ViT | Action $\\uparrow$ | Privacy $\\downarrow$ |\n|---|---|---|---|---|\n| Basic Model  | $\\surd$ |      | 66.1 | 66.9 |\n| Ours  | $\\surd$ |      |  **67.3**  | **53.9** |\n| Basic Model  |       | $\\surd$ | 66.1 | 71.2 |\n| Ours  |         |  $\\surd$   | **67.3** | **54.7** |\n\n[1] Dosovitskiy A, et al. An image is worth 16x16 words: Transformers for image recognition at scale, ICLR, 2021.\n\n\n\n\n>**Q6:** *\"The approach requires training data with labels for the private attributes available. If no such dataset is available, could the approach still be applied?\"**\n\n**A6:** Thanks for interesting question. Since previous works mostly use privacy attribute labels, we follow them also incorporate attribute labels in our approach. Facing the challenging situation where privacy labels are unavailable, here we propose a preliminary solution by introducing the unsupervised learning into our framework. Specifically, in our Basic Model, we can introduce an image reconstruction branch (which carries out unsupervised learning) to replace the privacy classification branch (which requires privacy labels).  By maximizing the image reconstruction loss and minimizing the action loss, we can encourage the anonymization model to remove visual content including privacy information, while remaining action clues. \n\nWe conduct experiments on HMDB51-VISPR. The results shown below demonstrate that this approach can enhance model's attribute-wise generalization performance even in the absence of privacy labels. Note that this is just a preliminary attempt, and we intend to further refine this approach in our future work.\n\n| Method  | Action $\\uparrow$ | Privacy $\\downarrow$ |\n|---|---|---|\n| Basic Model  | 44.5 | 54.0 |\n| Ours (unsupervised)  |  **44.8**  | **47.9** |"}, {"review_id": "tI1npxIGuU", "reviewer": "Reviewer_oDYm", "summary": "This paper deals with the privacy-preserving action recognition.\n\nThey propose a meta privacy-preserving action recognition (MPPAR) framework based on the concept of meta learning to improve both generalisation abilities w.r.t. novel privacy attributes & privacy attack models in a unified manner.\n\nThey simulate train/test shifts by constructing disjoint support/query sets w.r.t. the privacy attributes and attack models.\n\nThey conduct extensive experiments to show the effectiveness & generalisation of the proposed framework vs. existing methods.\n\n", "questions": "Please refer to the weakness section for the questions/concerns/issues to be addressed.\n\nAlso, below are some of the questions:\n\n- Figure 1 is not very clear to the reviewer e.g., why it shows some example attack models like MobileNet and ResNet, could you please explain the step clearly and sequentially?\n\n- Would skeleton-based action recognition methods solve the privacy issues? \n\n- What is the multiple binary cross-entropy loss? Could you please make the concept clear to readers?\n\n- What are the privacy attach models? How you split and choose the collection of privacy attach models for training?", "limitations": "Limitations of the proposed model is not provided in the submission.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ Overall the paper is well-written, technical sound and well-organised.\n\n+ The authors conduct enough experiments to show the effectiveness of the proposed model.\n\n+ There are some nice visualisations and comparisons provided in both main submission and the supplementary material.\n\n", "weaknesses": "- Formulas: In the approach section, the author gives a detailed description of every mathematical detail, which is worth encouraging, but neglects the main idea and intention behind the module design. To be honest, I am lost in a large number of mathematical symbols. I don't know why these modules are designed in this way and what the structure does.\n\n- It is suggested to have a notation section for the maths symbols used in the paper to make them clearer to readers. For example, regular fonts are scalars; vectors are denoted by lowercase boldface letters, matrices by the uppercase boldface, etc. \n\n- Figures: The research paper should use sufficient figures to show the details of the model, module details and experimental parts. In this paper, the author only draws one figure, and all the other details are contained in either texts or mathematical representations. This is a noteworthy shortcoming.\n\n- Description: the author uses too many explanations in the text. An excellent paper should reduce the proportion of auxiliary parts, make the organisation of paper clearer enough to readers. English expression should be improved.\n\n- Some typos and grammar mistakes, full stop is missing in Line 450, etc.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1:** *\"Formulas: the author eglects the main idea and intention behind the module design.\"*\n\n**A1:** Thank you for your suggestion. Below we provide a brief description to clarify the role of each component and the intention behind each design. (We will expand the following descriptions and add them into our main paper to improve the clarity of our method.) \n\nIn Section 3 of our main paper, we introduce the Basic Model that is used as the baseline in our approach. The Basic Model consists of three sub-models: action classification model $f^A$, privacy classification model $f^P$, and anonymization model $f^D$.\nAs shown in Figure 1 of the attached Rebuttal PDF file, the basic model contains two training procedures, \n\n(1) The anonymization model $f^D$ is trained to produce privacy-preserved videos that can fool the privacy classification model $f^P$ to predict incorect results. \n\n(2) The privacy classification model $f^P$ is trained to defense the fooling from the anonymization model $f^D$. \n\nThe training procedures of (1) and (2) are conducted in an adversarial competition manner. We repeat (1) and (2) to make these models become gradually stronger. During this process, the action classification model $f^A$ is also trained, aiming to ensure that the action model $f^A$ can assist the anonymization model $f^D$ to preserve the action information. After adversarial training, we can obtain a well-trained anonymization model that can effectively remove privacy information while preserving action clues.\n\nIn Section 4 of our main paper, we further propose a training scheme, namely *Virtual Training and Testing* scheme, to train the anonymization model $f^D$, aiming to further improve its generalization ability. Before training the anonymization model $f^D$, we partition the training data into a support set $D_s$ and a query set $D_q$ according to the privacy labels, **ensuring that these two sets contain totally different privacy attributes**. As shown in Figure 2 of the attached Rebuttal PDF file, the training of $f^D$ involves the following three steps:\n\n1. **Virtual training:** We train the anonymization model $f^D$ on the support set $D_s$.\n\n2. **Virtual testing:** We evaluate the trained anonymization model on the query set $D_q$. As $D_s$ and $D_q$ contains totally different attributes, the evaluation result can measure how well the model generalizes to new attributes. \n\n3. **Meta Optimization:** In this step, we treat the evaluation result (from **Virtual testing**) as a feedback to drive the model to adjust its training, enabling it to learn more generalizable knowledge.\n\n>**Q2:** *\"It is suggested to have a notation section for the maths symbols.\"*\n\n**A2:** Thanks for the reviewer's suggestion. We will add a separate section in the main text for symbol annotations.\n\n>**Q3 & Q6:** *\"Fig. 1 is not very clear. Why it shows some example attack models like MobileNet and ResNet?'\"*\n\n**A3 & Q6:** We have re-drawn the illustration of our proposed framework, adding more details and showing it step by step (see Figure 2 in the attached Rebuttal PDF file).  We use MobileNet and ResNet as examples to illustrate that different attack models are deployed for the support set $D_s$ and query set $D_q$.\n\n>**Q4 & Q5:** *\"Description and Typos.\"*\n\n**A4 & A5:** Thanks for valuable feedback. We will reduce the proportion of auxiliary parts and thoroughly check the entire manuscript.\n\n>**Q7:** *\"Would skeleton-based action recognition methods solve the privacy issues?\"*\n\n**A7:** While skeleton-based methods could handle some level of privacy concerns, the research regarding privacy preservation in RGB-based action recognition still remains important and practical. Here are the reason:\n\nIn many scenarios, besides action information, it is also necessary to incorporate information beyond action into the action recognition model. For instance, in the kitchen scenario [a] which contains many fine-grained actions, there are many actions that can be very similar, such as wash dishes and cut vegetables. Facing the similar human motion, accurately recognizing the action requires RGB camera to provide richer contextual information from the background, while simply using the skeleton to remove all other information will lead to performance degradation (as discussed in paper [b]).\n\n[a] Damen D, et al. Scaling egocentric vision: The epic-kitchens dataset, ECCV, 2018.\n\n[b] Sun Z, et al. Human Action Recognition from Various Data Modalities: A Review, TPAMI, 2023.\n\n>**Q8:** *\"What is the multiple binary cross-entropy loss?\"*\n\n**A8:** Here we first introduce the *Binary Cross Entropy Loss* which is used for classification of two classes. This binary classifier can use one output node, with Sigmoid activation function and labels take values 0, 1. Given the label $y$, the loss can be formulated as:\n$$L_{BCE} = -y\\log_{}{(p)},$$\nwhere $p$ is the classifier output.\n\nIn the PPAR task, besides the action label, each video also contains multiple privacy labels that are marked with binary value 0 or 1.\nTo address this multi-label classification, we adopt *Multiple Binary Cross-Entropy Loss*, in which, each attribute need to have a specific binary classifier to give a sigmoid output, which makes each prediction independent of other attributes. Then, we calculate the binary cross entropy loss for each attribute, and finally use the average loss value to represent the overall loss function. The loss function can be defined as:\n$$L_{MBCE} = -\\frac{1}{N} y_n\\log_{}{(p_n)},$$\nwhere $N$ is the number of attributes.\n\n>**Q9:** *\"What are the privacy attack models? How you split and choose the collection of privacy attack models for training?\"*\n\n**A9:** For fair comparison, we follow previous work [13] using 8 privacy attack models chosen from MobileNet-V2 family for training, and 10 different state-of-the-art privacy attack models such as ResNet-V1 and Inception-V1 for testing (see line103-107 of Appendix for more details)."}]

Author rebuttal:
We thank all reviewers for recognition of our contributions (Reviewer **hKip**: "this paper proposes a solution to address a important and interesting problem"; Reviewer **4pCF**: "the proposed method effectively defends against the privacy attack while maintaining the accuracy of action recognition"; Reviewer **EKAb**: "an interesting new avenue to PPAR research", "a practical defense against future attack approache", "the evaluation is extensive and the results are promising", "well-written and mostly easy to understand"; Reviewer **oDYm**: "enough experiments to show the effectiveness", "nice visualisations and comparisons"). 

Meta-review: This work proposes an action recognition framework that improves the privacy-preserving performance on both novel privacy attributes and novel privacy attack models based on the MAML meta-learning framework.

This paper got a borderline+ score (averaged 5.75), and all reviewers rated positively. During the rebuttal & discussion process, several concerns like wording, theory support, the necessity of privacy-preserving action recognition, and the experiments were raised by the reviewers. All Reviewers are actively engaged in the discussion with the authors and most of the concerns are well addressed.

The AC agrees with the reviewers that this work may have a positive impact on the less-explored privacy-preserving action regulation area.
The AC also notes the fact that some reviewers mentioned their limited familiarity with privacy-preserving learning. 

After careful consideration, The AC is inclined to accept this paper.
Decision: Accept (poster)