Federated Morozov Regularization for Shortcut Learning in Privacy Preserving Learning with Watermarked Image Data

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 09:58:14

Abstract: Federated learning is a promising privacy-preserving learning paradigm in which multiple clients can collaboratively learn a model with their image data kept local. For protecting data ownership, personalized watermarks are usually added to the image data by each client. However, the introduced watermarks can lead to a shortcut learning problem, where the learned model performs predictions over-rely on the simple watermark-related features and represents a low accuracy on real-world data. Existing works assume the central server can directly access the predefined shortcut features during the training process. However, these may fail in the federated learning setting as the shortcut features of the heterogeneous watermarked data are difficult to obtain.

In this paper, we propose a federated Morozov regularization technique, where the regularization parameter can be adaptively determined based on the watermark knowledge of all the clients in a privacy-preserving way, to eliminate the shortcut learning problem caused by the watermarked data. Specifically, federated Morozov regularization firstly performs lightweight local watermark mask estimation in each client to obtain the locations and intensities knowledge of local watermarks. Then, it aggregates the estimated local watermark masks to generate the global watermark knowledge with a weighted averaging. Finally, federated Morozov regularization determines the regularization parameter for each client by combining the local and global watermark knowledge. With the regularization parameter determined, the model is trained as normal federated learning. We implement and evaluate federated Morozov regularization based on a real-world deployment of federated learning on 40 Jetson devices with real-world datasets. The results show that federated Morozov regularization improves model accuracy by 11.22\% compared to existing baselines.

Main paper: 
 introduction: 
 With the growth of applying advanced multimedia technology to commercial applications, concerns about user data privacy have greatly increased [21] , and research on privacy-preserving learning has come into being. Federated learning [12, 50] emerges as a promising privacy-preserving learning paradigm, where multiple clients can collaboratively learn a model without exposing their private data to the central server. Federated learning has been widely adopted in many multimedia applications such as medical image classification [27] , anomaly detection in public safety surveillance [58] , and sentiment analysis in social media content [56] .For data ownership identification and copyright protection, digital watermarking technologies are developed and applied in many multimedia applications [13, 51] , through adding the well-designed digital watermark into the image data by the data owner [5, 20] . Training models with the watermarked data may lead to the shortcut learning problem, that is the learned model makes predictions based on the simple shortcut features in the training data, rather than learning the underlying complex core features of the target domain, and presents a good performance on the training dataset but decreased model accuracy on the unseen data [4, 28, 53] . For example, in medical image classification, a trained model detects pneumonia in chest X-rays (CXRs) relying on watermarks that represent which hospital the patient was seen instead of lung pathophysiology used by a radiologist [8, 53] .There are many works proposed to overcome the shortcut learning problem. According to where the shortcut feature is processed, existing works can be divided into data preprocesing-based [30, 34, 36, 42] and regularization-based [18, 32] methods. The data preprocessing-based methods assume the shortcut features of data are useless, and they eliminate the shortcut learning problem by detecting and removing the shortcut features from the training dataset. These methods may fail in learning with the watermarked data as the shortcut features (i.e., the watermark-related features) are important for data ownership identification, and cannot be directly removed in practice. For regularization-based methods, shortcut features are regularized based on certain prior knowledge during each training iteration. For example, FD [18] assumes the shortcut features are represented in specific frequency, and designs a feature-level regularization technique where a randomized filtering layer is applied after each convolution layer to prevent CNNs from learning frequency-specific imaging features. wMMD-T [32] assumes the causal Directed Acyclic Graph (DAG) indicating the relationship between the input image and output label is known and designs a regularizer that leverages knowledge of the causal DAG to efficiently learn a classifier. These regularization-based methods can work well in the centralized learning setting where the central server can directly obtain certain characteristics of the shortcut features. However, they may fail in a federated learning setting with watermarked data, as the characteristics of the watermark ZR:0 :DWHUPDUN+HWHUHRJHQHLW\'HJUHH )/7DVN$FFXUDF\)HG$YJ )'Figure 1 : The task accuracy of a model learned with various watermark heterogeneity.features of each client are uncertain and unknown to the server for privacy protection. Moreover, different clients may apply various digital watermarking techniques on the local data resulting in watermark heterogeneity, which further degrades the accuracy of the learned model. Our initial experiments show the impact of watermark heterogeneity under different regularization-based methods. As shown in Fig. 1 , with the watermark heterogeneity degree 𝛽 (detail setting can be seen in Sec. 4) increasing from 100 to 0.05, the accuracy of the learned model decreases up to 15.5% under all baselines.In this paper, we propose a federated Morozov regularization method to solve the shortcut learning problem of learning with watermarked data in a privacy-preserving way. Specifically, we first perform the local watermark mask estimation with the maximum a posteriori (MAP) method to generate the watermark mask, a matrix that can represent the characteristics of the watermarks. We observe that the embedded watermarks with various digital watermarking technologies can all be presented by the location and intensity map. Therefore, we estimate the watermark mask based on the distinct statistical distributions of natural images and artificial watermarks, capturing the divergence in their spatial and frequency domain characteristics. Then, we aggregate the estimated local watermark mask in the server to generate the global watermark mask with a weighted averaging model. Finally, we perform Morozov regularization-based local training by actively adjusting the regularization parameters with the estimated local and global mask. Intuitively, if the model training leads to worse overfitting to shortcut features, the regularization parameter will be increased, i.e., to aggressively mitigate overfitting introduced by the watermark; and vice versa. We evaluate federated Morozov regularization through experiments in real-world settings by deploying it on a test network of 40 Jetson devices, each with varying computational capabilities. We also evaluate our method on a real-world federated watermarked dataset, COVID-FL [52] , where watermark heterogeneity is present. Evaluation results demonstrate the superior performance of our method compared to the baselines. federated Morozov regularization improves the accuracy of the learned model by up to 11.22%. We also conducted an ablation study of federated Morozov regularization to validate the contribution of each component to FL model performance in watermarked datasets.The contributions of this paper can be summarized as:• We are the first to formulate the shortcut learning problem arising from watermarked datasets in federated learning and find that watermark heterogeneity can further degrade the learning performance.  
background & related work 2.1 fl for multimedia application: 
 The integration of federated learning (FL) with multimedia applications is fundamentally motivated by the need to safeguard privacy [23, 29, 55] . This approach has facilitated the advancement of multimedia applications involving personal data, such as image classification [27] , anomaly detection in public safety surveillance [58] , and sentiment analysis in social media content [56] . The bulk of current research in this area has been concentrated on tackling data-centric challenges [31, 57] , including non-iid data [26, 57] , data imbalance [44] , and the presence of noise [47] . However, a relatively unexplored issue in this domain is the influence of watermarked data in federated learning. Digital watermarking, a strategy widely adopted in multimedia applications for asserting data ownership [13, 51] and copyright protection [5, 20] , has found extensive application in data involving privacy and copyright issues, such as medical images [41] , surveillance videos [13] , and social media [38] . Despite its primary intent, watermarking unintentionally introduces detectable patterns into the data, precipitating a phenomenon known as shortcut learning.Shortcut learning refers to a phenomenon where deep learning models, during training, preferentially latch onto simple, detectable features-termed as shortcut features-instead of grappling with the more complex, core features of the data [10, 16] . This inclination can lead to models that perform well on training and in-distribution test data but falter significantly when faced with out-of-distribution inputs. Examples of shortcut learning include models relying on background elements [2] or specific textures for image classification [15] , and even the presence of watermarks [4, 8] .Solutions to shortcut learning have primarily focused on data prepossessing [30, 34, 36, 42] and regularization techniques [18, 32] . Data prepossessing often involve the removal of shortcut features [34, 36] or data augmentation [30, 42] . However, in federated learning scenarios, watermarks are added due to a lack of trust in the federated learning applications or to embed ownership directly into the training model, making their removal impractical. The regularization method often views shortcuts as a consequence of✓ = arg min ✓ X i2N p (i) f i (✓ (i) )Watermark Estimator FL ServerStep 1Local training Estimation mask < l a t e x i t s h a 1 _ b a s e 6 4 = " m e Q Q h y g U + i k N K + y Y W U D z H o a Y S L 4 = " > A A A B 7 3 i c b V D L T g J B E O z F F + I L 9 e h l I j H B C 9 k 1 v o 4 k X j x i I g 8 D K 5 k d Z m H C z O w 6 M 2 s k G 7 7 C g x c 1 X v 0 c j / 6 N A + x B w U o 6 q V R 1 p 7 s r i D n T x n W / n d z S 8 s r q W n 6 9 s L G 5 t b 1 T 3 N 1 r 6 C h R h N Z J x C P V C r C m n E l a N 8 x w 2 o o V x S L g t B k M r y Z + 8 5 E q z S J 5 a 0 Y x 9 Q X u S x Y y g o 2 V 7 p 6 6 w / u 0 z I 7 H 3 W L J r b h T o E X i Z a Q E G W r d 4 l e n F 5 F E U G k I x 1 q 3 P T c 2 f o q V Y Y T T c a G T a B p j M s R 9 2 r Z U Y k G 1 n 0 4 P H q M j q / R Q G C l b 0 q C p + n s i x U L r k Q h s p 8 B m o O e 9 i f i f 1 0 5 M e O m n T M a J o Z L M F o U J R y Z C k + 9 R j y l K D B 9 Z g o l i 9 l Z E B l h h Y m x G N g N v / u N F 0 j i p e O e V s 5 v T U t X N 0 s j D A R x C G T y 4 g C p c Q w 3 q Q E D A Mf Y F 7 k o W M Y G O l 2 6 f O 4 C 4 t 3 R + N O o W i W 3 Y n Q P P E y 0 g R M l Q 7 h a 9 2 N y K J o N I Q j r V u e W 5 s / B Q r w w i n o 3 w 7 0 T T G Z I B 7 t G W p x I J q P 5 0 c P E K H V u m i M F K 2 p E E T 9 f d E i o X W Q x H Y T o F N X 8 9 6 Y / E / r 5 W Y 8 M J P m Y w T Q y W Z L g o T j k y E x t + j L l O U G D 6 0 B B P F 7 K 2 I 9 L H C x N i M b A b e 7 M f z p H 5 c 9 s 7 K p 9 c n x Y q b p Z G D f T i A E n h w D h W 4 g i r U g I C A Z 3 i F N + f B e X H e n Y 9 p 6 4 K T z e z B H z i f P + f 3 k A 8 = < / l a t e x i t > x (j) kStep 2< l a t e x i t s h a 1 _ b a s e 6 4 = " 9 S a E L 9 9 p N r S b i n L 5 F b M F 5 2 g C f e I = " > A A A B / n i c b V B L S 8 N A G N z U V 6 2 v q O D F S 7 A I 9 V I S 8 X U s e P F Y w T 6 g j W G z 2 b Z L 9 x F 2 N 0 K J A X + L B y 8 q X v 0 b H v 0 3 b t o c t H V g 2 W H m + 9 j Z C W N K l H b d b 6 u 0 t L y y u l Z e r 2 x s b m 3 v 2 L t 7 b S U S i X A L C S p k N 4 Q K U 8 J x S x N N c T e W G L K Q 4 k 4 4 v s 7 9 z g O W i g h + p y c x 9 h k c c j I g C G o j B f Z B P x Q 0 U h N m r p R l w f g + r Z G T L L C r b t 2 d w l k k X k G q o E A z s L / 6 k U A J w 1 w j C p X q e W 6 s / R R K T R D F W a W f K B x D N I Z D 3 D O U Q 4 a V n 0 7 z Z 8 6 x U S J n I K Q 5 X D t T 9 f d G C p n K I 5 p J B v V I z X u 5 + J / X S / T g y k 8 J j x O N O Z o 9 N E i o o 4 W T l + F E R G K k 6 c Q Q i C Q x W R 0 0 g h I i b S o z H X j z P 1 4 k 7 d O 6 d 1 E / v z 2 r N t y i j T I 4 B E e g B j x w C R r g B j R B C y D w C J 7 B K 3 i z n q w X 6 9 3 6 m I 2 W r G J n H / y B 9 f k D 4 o C W Q Q = = < / l a t e x i t > m (i) k Model Aggregation < l a t e x i t s h a 1 _ b a s e 6 4 = " r n M G L n T u g J O W S q 0 0 d K q c O o i m I / M = " > A A A C e n i c b V H L a h s x F N V M 0 z Z N X 2 6 y L A Q R U 2 p T c G Z K 2 m Q T C L S L b h I S q J O A 5 R q N R m M r 0 W O Q 7 h S M M j + R P + u y f 9 J F F 9 X Y s 0 i c X h A 6 n H M u 9 5 W V U j h I k t 9 R / G j t 8 Z O n 6 8 8 2 n r 9 4 + e p 1 5 8 3 m u T O V Z X z I j D T 2 M q O O S 6 H 5 E A R I f l l a T l U m + U V 2 / a X R L 3 5 y 6 4 T R 3 2 F e 8 r G i U y 0 K w S g E a t K 5 J Z m R u Z u r 8 P n j G h 9 i 4 i o 1 8 Y I I T R S F G a P S n 9 Q 1 J p I X 0 M P + x n + t f / i e 6 N c 3 9 a 5 f m q 9 W z a 3 p q j F h w n I D m G T U e t e m Y m L F d A Z 9 f L e 6 a s V J p 5 s M k k X g h y B t Q R e 1 c T r p / C K 5 Y Z X i G p i k z o 3 S p I S x p x Y E k 7 z e I J X j J W X X d M p H A W q q u B v 7 x e 5 q / C 4 w O S 6 M D U 8 D X r B 3 M z x V r m k w O J s J 3 a r W k P / T R h U U B 2 M v d F k B 1 2 x Z q K g k B o O b Q + B c W M 5 A z g O g z I r Q K 2 Y z a i m D c K 6 w g 3 R 1 4 o f g / O M g / T z 4 d L b X P U r a b a y j t 2 g H 9 V C K 9 t E R + o Z O 0 R A x 9 C f a j t 5 H v e h v v B P 3 4 w 9 L a x y 1 O V v o X s R 7 / w A 1 v M N Y < / l a t e x i t > M = X i2N 0 @ |D (i) |/ X j2N |D (j) | • s(i) 1 A m (i) < l a t e x i t s h a 1 _ b a s e 6 4 = " V l 3 R V 8 + k z J x 4 i P i R r v n L 1 i G V i t I = " > A A A B 9 X i c b V D L S g M x F L 1 T X 7 W + q i 7 d B I v g q s y I r 2 X R j R u h g n 1 A O 5 Z M J t O G Z p I h y S h l 6 H + 4 c a G I W / / F n X 9 j p u 1 C W w + E H M 6 5 l 5 y c I O F M G 9 f 9 d g p L y y u r a 8 X 1 0 s b m 1 v Z O e X e v q W W q C G 0 Q y a V q B 1 h T z g R t G G Y 4 b S e K 4 j j g t B U M r 3 O / 9 U i V Z l L c m 1 F C / R j 3 B Y s Y w c Z K D 9 1 A 8 l C P Y n t l t + N e u e J W 3 Q n Q I v F m p A I z 1 H v l r 2 4 o S R p T Y Q j H W n c 8 N z F + h p V h h N N x q Z t q m m A y x H 3 a s V T g m G o / m 6 Q e o y O r h C i S y h 5 h 0 E T 9 v Z H h W O f R 7 G S M z U D P e 7 n 4 n 9 d J T X T p Z 0 w k q a G C T B + K U o 6 M R H k F K G S K E s N H l m C i m M 2 K y A A r T I w t q m R L 8 O a / v E i a J 1 X v v H p 2 d 1 q p X c 3 q K M I B H M I xStep 3 model overparameterization [32] . Techniques like FD [18] emphasize high-frequency shortcut features, while methods like wMMD-T [32] focus on background elements as shortcut features. Yet, these shortcut features do not align with those introduced by watermarks. Moreover, these approaches often require prior knowledge of the shortcut features from client data, such as labels or filter parameters, which contradicts the privacy-preserving nature of federated learning. Utilizing global information from the server side also fails to address the challenges brought by watermark heterogeneity.< l a t e x i t s h a 1 _ b a s e 6 4 = " d E y W k A / 3 F l + 7 X D Q l M y y W C g T 7 X s Q = " > A A A B / 3 i c b V A 7 T 8 M w G H T K q 5 R X A I m F J a J C K k u V I F 5 j B Q t j k e h D a k P k O E 5 r a s e R 7 S B V I Q N / h Y U B h F j 5 G 2 z 8 G 5 w 2 A 7 S c Z P l 0 9 3 3 y + f y Y E q l s + 9 s o L S w u L a + U V y t r 6 x u b W + b 2 T l v y R C D c Q p x y 0 f W h x J R E u K W I o r g b C w y Z T 3 H H H 1 3 l f u c B C 0 l 4 d K v G M X Y Z H E Q k J A g q L X n m X t / n N J B j p q + U Z d 7 o L q 3 d H 2 W e W b X r 9 g T W P H E K U g U F m p 7 5 1 Q 8 4 S h i O F K J Q y p 5 j x 8 p N o V A E U Z x V + o n E M U Q j O M A 9 T S P I s H T T S f 7 M O t R K Y I V c 6 B M p a 6 L + 3 k g h k 3 l E P c m g G s p Z L x f / 8 3 q J C i / c l E R x o n C E p g + F C b U U t / I y r I A I j B Q d a w K R I D q r h Y Z Q Q K R 0 Z R V d g j P 7 5 X n S P q 4 7 Z / X T m 5 N q 4 7 K o o w z 2 w Q G o A Q e cMorozov regularization [39] is one type of tool to adjust regularization parameters actively. One key principle of these methods is the discrepancy principle [43] . The rationale is that for a good regularized solution, the norm of the residual should match the noise level of the data.Morozov regularization has been used in many applications in the past, e.g., to regularize noises from satellite sounder measurements for atmospheric profiling applications [24] , to regularize sensor noises in digital images [6] and machine learning [19, 39] recently.The suitability of Morozov regularization for our problem lies in its precision in targeting specific distributions or explicitly formulated noise, offering localized regularization rather than a blanket, global approach. This characteristic is particularly aligned with the challenges posed by watermarks, which introduce shortcut features localized within parts of an image, rather than affecting it uniformly. Unlike other regularization methods that might operate under broad assumptions about noise or apply regularization uniformly across the entire data set, Morozov regularization provides an adaptive mechanism to fine-tune the regularization parameter, thereby mitigating the shortcut learning effect.As compared to other regularization, Morozov regularization is simple and has less assumptions on noise approximation [1] , practical a-posteriori rules [35] , and/or convergence rate [37] . We choose Morozov regularization for its widely applicability and leave other types of regularization into future works. 
federated morozov regularization 3.1 problem definition: 
 Federated Learning leverages a set of distributed clients N = {1, . . . , 𝑁 } to iteratively learn a global model 𝜽 without leaking any private local data to the central server [33] . In each client 𝑖, the local dataset is defined as 𝐷 (𝑖 ) . For data ownership identification, each data sample 𝑥 The integration of digital watermarks into image data for ownership identification introduces several challenges in the federated learning environment. Firstly, accurately modeling the embedded watermarks (𝑛 (𝑖 ) 𝑘 ) within the data requires sophisticated techniques to distinguish and quantify their impact on the learning process. Secondly, the distributed nature of federated learning complicates the task of addressing the variability in watermark characteristics across different clients while maintaining global model performance. Thirdly, it is crucial to mitigate the performance degradation caused by watermarks without compromising the privacy of the data.We propose a federated Morozov regularization method to address these challenges, and Fig. 2 Our stochastic approach is based on maximum a posteriori (MAP) estimation, as described in [48] . Consider the classical problem of watermark embedding, which involves embedding a watermark into an image without considering the image content. In the most general form in communication codec theory [49] , the process of 𝑊 𝑀 (•) can be modeled as 𝑥 ′ = 𝑥 + 𝑛, where 𝑥 ′ represents the watermarked data, 𝑥 is the original data, 𝑥 ∈ R 𝑁 with 𝑁 = 𝑀 × 𝑀, and 𝑛 denotes the watermark. Our goal is to estimate n, which is an approximation of the watermark 𝑛.Under the general assumption [49] , we model the watermark as a Gaussian random variable. Let watermark sample 𝑛 𝑢,𝑣 (1 ≤ 𝑢, 𝑣 ≤ 𝑀) and image sample 𝑥 𝑢,𝑣 (1 ≤ 𝑢, 𝑣 ≤ 𝑀) be defined on the vertices of a grid 𝑀 × 𝑀. Further, let all samples be independent and identically distributed, we have conditional probability density of 𝑛 𝑢,𝑣 :EQUATIONwhere the 𝜎 𝑛 𝑢,𝑣 of the watermark in the (𝑢, 𝑣) location signifies its intensity, Δ 𝑛 = 𝑥 𝑢,𝑣 -𝑛 𝑢,𝑣 . Higher variance indicates a more noticeable watermark (albeit with possible image distortion), whereas lower variance results in a subtler watermark.To estimate the watermark throughout an image, we use a local estimation mask 𝒎 = [ n𝑢,𝑣 ] 1≤𝑢,𝑣 ≤𝑀 , which is a matrix represent each client's watermark information in local dataset. The index (𝑢, 𝑣) in this mask represents the watermark location. Each n𝑢,𝑣 is determined by the MAP criterion:EQUATIONwhere ñ represents a hypothetical watermark value being considered during the optimization process to maximize the posterior probability. The estimation accuracy enhancement is due to MAP estimation's statistical convergence towards the true watermark distribution as the dataset grows.In FL environments, the aggregation of local models is a crucial step for synthesizing a global model that benefits from the distributed learning process. Analogously, the aggregation of local watermark estimation masks is essential for constructing comprehensive global watermark knowledge.The aggregation of the global watermark estimation mask, denoted by 𝑴, incorporates contributions from local watermark estimation masks 𝒎 (𝑖 ) from each client 𝑖 within the network N . The aggregation process is governed by the equation:EQUATIONwhere the weight for each client's local mask 𝒎 (𝑖 ) is determined by the product of two key factors. The first factor,|𝐷 (𝑖 ) | 𝑗 ∈N |𝐷 ( 𝑗 ) |, consider the relative data sample size |𝐷 (𝑖 ) | of the 𝑖-th client, indicating the proportion of data contributed by this client in comparison to the total data volume across all clients in N . The second factor, s (𝑖 ) , corresponds to the average size of the watermark estimation mask for the 𝑖-th client, which is computed as the mean of the dimensions of the mask 𝒎 (𝑖 ) . This measure reflects the spatial extent of the watermark information present within the client's data.After obtaining the global watermark estimation mask 𝑴, it is a mask integration with the local masks to refine the watermark knowledge for each client. The refined local mask for client 𝑖, denoted by 𝒎 * (𝑖 ) , is achieved by blending 𝑴 with 𝒎 (𝑖 ) :EQUATIONwhere 𝛽 (𝑖 ) ∈ [0, 1] is an adaptive hyperparameter that controls the degree to which the global mask influences the refined local mask. The value of 𝛽 (𝑖 ) is dynamically adjusted based solely on the training performance difference, ΔAcc (𝑖 ) , which is the difference between the highest validation accuracy among all clients and the validation accuracy of the current client 𝑖. To ensure 𝛽 (𝑖 ) scales appropriately between 0 and 1, it is calculated as follows:EQUATIONThis formula uses an exponential function to decrease the influence of ΔAcc (𝑖 ) as it increases, ensuring that 𝛽 (𝑖 ) remains within the desired range and effectively balances the contribution of the global mask based on the relative performance of each client. Regularization adds a term 𝑟𝑒𝑔(•) to the loss function 𝑓 (𝜽 ; 𝑥, 𝑦), comprising a parameter matrix 𝜶 and norm 𝑅(𝜽 ), formulated as 𝑟𝑒𝑔(𝜽 ) = 𝜶 𝑅(𝜽 ). The matrix 𝜶 balances regularization's importance, with higher values increasing bias and reducing overfitting, and lower values doing the opposite. This balance is captured by 𝜶 = 𝜶 (𝛿), where 𝛿 measures deviation from real data.Mathematically, refer to [25] , the loss function with regularization is formulated as below,EQUATION)Morozov regularization is a principle for choosing a regularization parameter, i.e., 𝜶 , to stabilize the machine learning model to be trained. Specifically, let 𝑥 𝛼 beEQUATION𝜶 can be considered as a control parameter. If 𝜶 is too small, the model overfits the watermarked in the data; and if 𝜶 is too big, the model loses the essential details. If 𝑦 𝛿 is the watermarked data and assume that 𝛿 is the known noise level introduced by the watermarked data, then 𝜶 is chosen such that:EQUATIONIn other words, Morozov regularization chooses the value of 𝜶 that can make the norm || • || equal to the noise level (also called the Morozov's discrepancy principle [43] ).The federated Morozov regularization for FL in Alg. 1 operates in three main phases: watermark estimation, mask aggregation, and regularization parameter computation. Initially, each client's model parameters 𝜽 (𝑖 ) are initialized. The watermark estimation phase involves using MAP-based method to estimate the n(𝑖) 𝑘 in each data point of client 𝑖's dataset 𝑑 (𝑖 ) and get the 𝒎 (𝑖 ) (Lines 4-7).The watermark aggregation use the clients' watermark estimation mask to aggregate a global mask with Eq. ( 4) before federated learning training process (Lines 10-11).During the federated learning process, each client refines the watermark estimation mask as outlined in Lines 14-17. In this phase, the learning module of each client employs Morozov regularization to compute the regularization parameters. This involves setting an initial discrepancy tolerance and 𝜶 (𝑖 ) 𝑘 , which are iteratively refined based on model predictions ŷ (𝑖 ) 𝑘 , residuals, and discrepancy measures until they converge within the set tolerance (Lines 18-24).Finally, the algorithm utilizes the refined 𝜶  
evaluation 4.1 evaluation settings: 
 We assess the performance of our technique in a client-server testbed that we have constructed. The server is equipped with an Nvidia RTX 3090 GPU and an AMD Ryzen 9 5900X CPU, running on Ubuntu 20.04 LTS. For client devices, we employ 2 × NVIDIA Jetson AGX Orin, 3 × Jetson Orin Nano 8GB, 5 × Jetson Orin Nano 4GB, and 30 × Jetson Nano. The performance detail can be seen in Table . 1. Our testbed setting and equation 1 have been shown in Fig. 3 to understand the efficacy of federated Morozov regularization in heterogeneity edge clients. We connected each client device to switches via an Ethernet cable. Data exchange in federated learning, including metadata and models, is facilitated by accessing the 1 The 4GB and 8GB Jetson Orin Nano boards have the same appearance. Upload 𝒎 (𝑖 ) Update 𝛽 (𝑖 ) using Eq. ( 6);17Refine 𝒎 * (𝑖 ) using Eq. ( 5); Models and datasets. In the experimental phase, our investigation employed a range of neural network architectures to perform the image recognition task. The first is a Lite-CNN 2 , characterized by its simplicity yet effectiveness. Alongside our custom CNN, we integrated two well-established models: VGG [45] and ResNet-18 [14] .The initial learning rate is set to 0.1, and the batch size is set to 64 by default. Using the SSP [17] synchronization strategy, the local epochs are set to 5 by default. In our experimental setup, we evaluated the performance of our proposed method using widely recognized image classification datasets, including MNIST [9] , Cifar-10 [22] , Tiny-ImageNet [7] and COVID-FL [52] . In our experimental design, MNIST was trained using the Lite-CNN to assess the effectiveness of our method on simple tasks with a straightforward model. For Cifar-10, we employed the VGG model to train, leveraging its depth and complexity for a more detailed image classification task. The Tiny-ImageNet, serving as a multi-classification challenge, and the binary classification task of COVID-FL were both approached using the ResNet architecture, to handle the complexity and scale of these datasets adequately. Watermark setting. In the creation of our watermarked dataset, various watermark embedding techniques, including frequency watermark: DWT, DCT, DFT, LSB, and spatial watermark: LSB, explicit watermarking [3, 41] , are employed. Although the adjustment parameters for watermark intensity vary across different methods (for instance, the intensity in explicit watermarks refers to transparency, while in some frequency domain watermarks, like DFT, it refers to modulation amplitude), we normalize the intensity of all watermarks to a 0-1 scale. The embedding location in spatial domain watermarks denotes the position of the watermark within the image (such as the center or edges), whereas in frequency domain watermarks, it refers to the frequency within the image spectrum (like high, mid, or low frequency; in DCT, this ranges from the LL to HH domain).The watermark embedding intensity is adjusted between 0.01 and 1 for our experiments. Each client in the FL employs the same watermarking method and parameters, ensuring consistency across the dataset. Evaluation metrics. Our performance evaluation focuses FL performance. Referring to the evaluation metric of the shortcut learning research [18, 32, 36] , FL performance is assessed using task accuracy, which measures the percentage of correct predictions by the FL models on a distributed dataset, and loss, indicating the prediction error with lower values signifying better performance. Benchmark methods. We compare the federated Morozov regularization with the following peer robust training methods in 2 Lite-CNN comprises two convolutional layers, each with a 5x5 kernel size and 64 channels, succeeded by a 3x3 max pooling layer. The network also includes two fully connected dense layers, the first containing 384 units and the second 192 units, culminating in a softmax output layer for classification.FL [40] [11] , generalized regularization [54] and regularization for shortcut learning [18] .• FedAvg [33] : Used to establish a performance baseline in our experiments, serving as a foundation for comparison with other FL algorithms.• GroupLasso [54] : a generalized regularization for machine learning by adding a penalty term. We modified GroupLasso to federated learning version based on the client-level profiling setting.• AFL [11] : Using global model transmission, local gradient calculations, and averaging, with hyperparameters set to 𝛼 1 = 0.75, 𝛼 2 = 0.01, 𝛼 3 = 0.1 in our experiments.• RFA [40] : A Roubstness aggregation method for corrupted data. We applied with hyperparameters as per the original paper: R = 3 and 𝑣 = 10 -6 . • FD [18] : A feature regularization with frequency filter tools.We modified FD to federated learning version (Fed-FD) based on the client-level profiling setting. The experimental design, as outlined in Table 2 , bifurcates the analysis into two primary scenarios: inference on clean data and inference under watermarked conditions. This distinction aims to uncover the impact of shortcut learning induced by watermarks, which affects not only the inference with watermarked features but also the performance on clean data, highlighting the pervasive influence of watermarks on model behavior. The settings for data and watermark heterogeneity are set to 𝛼 = 0.5, 𝛽 = 0.5, which be defined in Sec. 4.2.2.Our method demonstrates superior accuracy across all datasets and settings, underscoring its effectiveness in mitigating the adverse effects of shortcut learning in FL. Specifically, in the clean inference setting, our approach achieves an accuracy of 97.35% on MNIST, 80.86% on Cifar-10, 35.43% on ImageNet, and 87.14% on COVID-FL. These results are notably higher than those obtained with other methods, such as FedAvg, GroupLasso, AFL, RFA, and Fed-FD. The improvement is even more pronounced in the watermarked dataset & inference setting, with scores of 95.24% on MNIST, 79.26% on Cifar-10, 33.29% on ImageNet, and 84.10% on COVID-FL. The detail learning performance with epoch growing can be seen in Fig. 4 .The underperformance of other methods can be attributed to their inability to effectively address the dual challenge posed by non-IID data and the presence of watermarks. Methods like FedAvg and GroupLasso, while foundational in FL, lack specific mechanisms to counteract the nuanced effects of watermarked data, leading to compromised accuracy. AFL and RFA, despite introducing robustness in aggregation, do not directly tackle the issue of shortcut learning induced by watermarks. Fed-FD, which applies feature regularization, shows promise but still falls short of fully mitigating the impact of watermarks on model learning. the presence of non-IID (independent and identically distributed) data challenges. For the non-IID problem in the FL experiment, we define the degree of non-IID data and non-IID watermark as follows:In a multi-client training scenario, each client's data is independently sampled with class labels from 𝑁 classes, following a categorical distribution with vector 𝑞 (𝑞 𝑖 ≥ 0, 𝑖 ∈ [1, 𝑁 ], ∥𝑞∥ 1 = 1). Non-IID client data is simulated by sampling 𝑞 from a Dirichlet distribution, Dir(𝛼p), where p is the prior class distribution, and 𝛼 > 0 determines client similarity. An infinite 𝛼 implies uniform client distributions, while 𝛼 near zero results in maximum divergence among clients.In the context of non-IID watermark settings, we adopt a distribution similar to the Dirichlet distribution to manage the variability in watermark characteristics such as intensity (𝐼 ) and location (𝐿). Intensity ranges from 0 (no watermark) to 1 (maximum intensity), while location varies from low-frequency areas or image edges to high-frequency areas or central regions. We introduce a parameter 𝛽 in Dir(𝛽p) to control the degree of non-IID in the watermark distribution. A higher 𝛽 indicates more uniformity in watermark characteristics across clients, leading to similar intensity and location settings. Conversely, a lower 𝛽 results in greater diversity, with each client having distinct watermark intensity and placement. This approach allows us to simulate a spectrum of watermark patterns across different clients, reflecting various degrees of intensity and placement. Unlike other datasets, for the real-world dataset COVID-FL, the data is already divided among different clients by medical institutions, thus we utilize the official non-IID configuration distribution to proceed. The variation in equipment used by different medical institutions, along with their respective watermark design preferences, inherently introduces a non-IID distribution of watermarks. Therefore, COVID-FL, as a more realistic watermarked dataset, can be considered a reference for real-world issues and does not require additional non-IID watermark design and settings.In our experimental analysis, the combined impact of non-IID data and non-IID watermark on the federated Morozov regularization technique is depicted through heatmaps, revealing a compounded decrease in accuracy with the simultaneous presence of both non-IID conditions. We have selected Fed-FD as the benchmark for testing our method based on its superior performance as demonstrated in Sec. 4.2.1. When the non-IID degree for both data and watermark is at its highest, we observe a notable reduction in accuracy, illustrating the challenges posed by these conditions. For example, with 𝛼 of 0.5 and 𝛽 of 0.5, the accuracy drops to around 68.04%. Modifications to the technique, as reflected in the second heatmap, show improvements in this challenging scenario with a notable increase in accuracy. Under the same high non-IID conditions, the accuracy improves to 79.26%. The third heatmap, which focuses on the percentage of improvement, highlights the effectiveness of our modifications. In scenarios with non-IID data and watermark, our method achieves a substantial improvement, with the most pronounced increase in accuracy reaching up to 11.22%.We present an analysis of three components designed for such environments: MAP-based watermark estimation (MAP), watermark estimation aggregation (Aggr.) and Morozov regularization (Moro.) in Table . 3. The goal is to evaluate how effectively these components, can counteract the reduction in accuracy often caused by watermarking, compared to alternative methods or variations. Study on watermark estimation. Our exploration delved into the efficacy of MAP-based watermark estimation by comparing it against both its variants and analogous statistical methodologies. This included the Blind Image Quality Measurement (denoted 'DWDKHWHURJHQHLW\GHJUHH :DWHUPDUNKHWHURJHQHLW\GHJUHH as Blind) [46] , a technique predicated on leveraging statistical attributes to gauge image quality, and the strategy of stacking all dataset images to generate a uniformly weighted mask, tantamount to an averaged weighted MAP-based estimation (denoted as Stacking). As delineated in Table 3a , the MAP approach manifested a notably superior accuracy enhancement relative to its counterparts, with a 12.68% increment over Stacking within the Cifar-10 dataset. Such findings underscore that methodologies centered on image quality estimation (Blind) and indiscriminate estimation of images and watermarks (Stacking) are ineffectual in procuring a robust watermark estimation. Study on estimation mask aggregation. We delve into the efficacy of watermark mask aggregation by both omitting this component (denoted as w/o.) and evaluating its variants, specifically average aggregation (denoted as Avg.), where the local masks from all clients undergo aggregation with equal weighting. As evidenced in the Table. 3b, aggregation demonstrates enhanced performance in the Cifar-10 dataset, characterized by strong heterogeneity and a smaller quantity of images. Conversely, for datasets with a larger volume and more uniform data, such as Tiny-ImageNet and COVID-FL, the performance difference compared to average aggregation is minimal. This phenomenon can be attributed to the intrinsic purpose of mask aggregation, which is to furnish a global mask that aids clients with less data in obtaining a more applicable mask. Therefore, if the local datasets of clients are sufficiently large, the improvement brought about by aggregation may be marginal.Study on Morozov regularization. In our ablation study focusing on Morozov regularization, we maintained identical inputs for the estimation mask while employing a simplified form of regularization. Morozov regularization, conceptualized as a variant of Tikhonov regularization, introduces parameter adjustments that are more finely tuned to the noise levels encountered. Thus, Tikhonov regularization (denoted as Tik.) is utilized as a comparative measure to ascertain the significance of adjustments in regularization parameters. Furthermore, we investigate whether L1 regularization, a widely referenced regularization technique, also demonstrates improvements in the context of prior information on watermark estimation (denoted as L1). Insights from Table . 3c reveal that the enhancements attributed to Morozov Regularization are predominantly observed in datasets with smaller capacities, such as Cifar-10, and in datasets where the watermark patterns are relatively fixed, such as COVID-FL. It is also observed that, although other forms of regularization exhibit limited improvements over the baseline, their compatibility with watermark estimation is not as pronounced. 
conclusion: 
 Our  
 
 Appendix:  


Decision: Summary:

This paper proposes a federated Morozov regularization technique to address shortcut learning from watermarked data in federated learning. The method was tested on 40 Jetson devices and demonstrated significant improvements over existing baselines.

Strengths:
1. The paper is well-organized and clearly presents the proposed method.
2. The motivation for addressing shortcut learning in federated learning using watermarked data is strong, and the idea is interesting.
3. Experimental results show that the proposed method outperforms existing techniques.

Limitations:
1. The paper should provide more details about the digital watermarking techniques used and justify the use of maximum a posteriori (MAP) estimation for watermark masks.
2. The authors should include information about the datasets' volume and characteristics, and justify their choice.
3. The evaluation should consider the impact of non-i.i.d data and watermarking across different datasets, not just CIFAR10. The use of different neural network models for different datasets should be explained.

After the rebuttal, one reviewer raised the final rating, so all reviewers tend to accept this submission. Therefore, I will recommend this paper as Accept (Poster).

Reviews: review: paper_summary: paper_summary: This paper proposes a federated Morozov regularization technique to address shortcut learning from watermarked data in federated learning. Tested on 40 Jetson devices, this method surpasses existing baselines by a large margin.
strength_weakness: strength_weakness: 1. This paper is well-organized and clear.
2. The motivation is good and the idea is interesting.
3. Experimental results are promising.
limitations: limitations: As I am not very familiar with this field, could you provide a detailed explanation of how this paper relates to multimedia?
paper_summary: paper_summary: The paper discusses a method to improve federated learning, a collaborative model training approach that keeps data local to ensure privacy. In federated learning, clients often add personalized watermarks to their image data to protect data ownership. However, these watermarks can create a shortcut learning problem where the model relies too heavily on the watermark features, reducing its accuracy on real-world data.

To address this issue, the authors propose a federated Morozov regularization technique. This method adaptively determines a regularization parameter based on the watermark information from all clients in a privacy-preserving manner. The process involves:

- Local Watermark Mask Estimation: Each client estimates the locations and intensities of their watermarks.
- Global Watermark Knowledge Aggregation: The estimated local watermark masks are aggregated using weighted averaging to generate global watermark knowledge.
- Regularization Parameter Determination: Combining local and global watermark knowledge, the regularization parameter for each client is determined.

With this parameter, the model is trained as in standard federated learning. The method was implemented and evaluated on a real-world deployment with 40 Jetson devices and real-world datasets. Results show that federated Morozov regularization improves model accuracy by 11.22% compared to existing methods.
strength_weakness: strength_weakness: - The paper provides detailed architectural diagrams, mathematical derivations, and algorithms to illustrate the methodology's process.
- The experimental section in the paper is clear and well-explained, including relevant photos of the experimental machines to ensure the authenticity of the experiments.
- The evaluation section presents data clearly, demonstrating the proposed method's advantages in improving accuracy and training speed.
- The article includes an ablation study, examining the potential impacts of watermark estimation, estimation mask aggregation, and Morozov regularization on the experiments.
limitations: limitations: - This paper focuses on the optimization of federated shortcut learning and watermarked data, with relatively weaker contributions to aspects such as transmission, processing, and presentation of multimedia and unimedia data.
- In the Introduction section, page 2, line 141, the content is "increasing from 100 to 0.05," but the actual is "increasing from 100 to 0.5."
- In the sentence, "Moreover, different clients may apply various digital watermarking techniques on the local data resulting in watermark heterogeneity, which further degrades the accuracy of the learned model," the authors should list the specific digital watermarking techniques used as background knowledge for readers.
- The authors should add the reason for using maximum a posteriori (MAP) estimation for the watermark mask instead of other methods.
- In the dataset selection, the authors should include information about the data volume and characteristics, such as the number of images, image sizes, and label types, and explain the significance of choosing these datasets.
- The authors used different neural network models for different datasets. Although the intent might be to demonstrate the general applicability of federated Morozov regularization, this approach could impact the results when using models of different scales. If possible, an ablation study should be added to address this.
- Issues with Figure 5:
    - The font size of the percentage numbers is too large, causing some overlapping and affecting readability.
    - The figure's location is not indicated in the text.
    - The three subplots do not have a unified color scheme, nor is there an explanation for the color choices.
    - In caption (c): "Improvement between 5a and 5b," the text does not explain what 5a and 5b are.
paper_summary: paper_summary: This paper proposes a Federated Morozov Regularization method to address the shortcut problem caused by watermarked data in federated learning scenarios. It models the watermark using Gaussian random variable and estimates the watermark with position and density information based on maximum a posteriori estimation. The server aggregates the local watermark information uploaded by clients to obtain global watermark information. Clients then adaptively generate regularization parameters based on the received global watermark information and local watermark information for model training. The paper effectively eliminates the shortcut problem caused by watermarks in federated learning while preserving privacy, performing well under both non-IID data and watermark conditions.
strength_weakness: strength_weakness: -	Novel approach: The authors address the shortcut problems introduced by watermark data in federated learning scenarios and introduce Morozov Regularization method for diverse watermark techniques to mitigate their impacts on model performance.
-	Theoretical statement: The author defines the problem being addressed and provides a detailed mathematical formulation of the three key steps, facilitating a comprehensive understanding of the design.
-	Adequate evaluation and promising results: The authors conduct extensive experiments using four datasets, considering various watermark embedding techniques and the heterogeneity of both data and watermarks. The evaluation results demonstrate that the proposed method outperforms baselines and performs well under non-IID data and watermark conditions.
-	The paper is well-written and easy to follow.
limitations: limitations: -	Watermark setting: The paper experiments with various watermarking techniques， but appears to only consider diversity in intensity and position of the same watermark. However, in real-world scenarios, participants may utilize different watermarking methods among themselves and internally. The authors should provide reasons for focusing solely on diversity within the same watermarking method and further specify the scope of the paper.
-	Experiments: The experimental design could be expanded to include three primary scenarios, leveraging clean datasets to better elucidate the impact of watermarked data through inference.
-	Writing errors: The "return" statement in line 26 of the algorithm 1 appears unnecessary and may cause subsequent statements not to be executed. The phrase "Our introduces" in the beginning of the conclusion lacks a subject.
paper_summary: paper_summary: This paper proposes a federated Morozov regularization technique, where the regularization parameter can be adaptively determined based on the watermark knowledge of all the clients in a privacy-preserving way, to eliminate the shortcut learning problem caused by the watermarked data. Specifically, federated Morozov regularization firstly performs lightweight local water mark mask estimation in each client to obtain the locations and intensities knowledge of local watermarks. Then, it aggregates the estimated local watermark masks to generate the global watermark knowledge with a weighted averaging. Finally, federated Morozov regularization determines the regularization parameter for each client by combining the local and global watermark knowledge. With the regularization parameter determined, the model is trained as normal federated learning.
strength_weakness: strength_weakness: 1.	This paper is the first to formulate the shortcut learning problem arising from watermarked datasets in federated learning and find that watermark heterogeneity can further degrade the learning performance. 
2.	This paper proposes federated Morozov regularization, a new regularization method that can automatically adjust the regularization parameters based on the watermark knowledge of all clients in a privacy-preserving way. 
3.	The authors evaluate federated Morozov regularization by deploying a real-world testbed of 40 Jetson devices with diverse computational capacities and comparing it to several baselines with real-world datasets.
limitations: limitations: 1.	Evaluate the impact of varying degrees of Non-i.i.d data and watermarking across different datasets on the experimental results. Assessing only CIFAR10 is not sufficient.

2.	Why use VGG for CIFAR10 and ResNet for Tiny-ImageNet? ResNet also performs well on CIFAR10.