ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition
Authors: Joseph Fioresi
Ishan Rajendrakumar Dave
Mubarak Shah
Venue: ICLR 2025 Poster (2025)
Keywords: Bias Mitigation
Action Recognition
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose an effective adversarial framework for mitigating static biases in action recognition models without requiring prior knowledge of bias attributes.

Abstract: Bias in machine learning models can lead to unfair decision making, and while it has been well-studied in the image and text domains, it remains underexplored in action recognition. Action recognition models often suffer from background bias (i.e., inferring actions based on background cues) and foreground bias (i.e., relying on subject appearance), which can be detrimental to real-life applications such as autonomous vehicles or assisted living monitoring. While prior approaches have mainly focused on mitigating background bias using specialized augmentations, we thoroughly study both foreground and background bias. We propose ALBAR, a novel adversarial training method that mitigates foreground and background biases without requiring specialized knowledge of the bias attributes. Our framework applies an adversarial cross-entropy loss to the sampled static clip (where all the frames are the same) and aims to make its class probabilities uniform using a proposed entropy maximization loss. Additionally, we introduce a gradient penalty loss for regularization against the debiasing process. We evaluate our method on established background and foreground bias protocols, setting a new state-of-the-art and strongly improving combined debiasing performance by over 12% absolute on HMDB51. 
Furthermore, we identify an issue of background leakage in the existing UCF101 protocol for bias evaluation which provides a shortcut to predict actions and does not provide an accurate measure of the debiasing capability of a model. We address this issue by proposing more fine-grained segmentation boundaries for the actor, where our method also outperforms existing approaches.

Published as a conference paper at ICLR 2025
ALBAR: ADVERSARIAL LEARNING APPROACH TO MIT- IGATE BIASES IN ACTION RECOGNITION
Joseph Fioresi, Ishan Rajendrakumar Dave, Mubarak Shah
Center for Research in Computer Vision, University Central Florida, Orlando, USA (joseph. fioresi, j dave ducf edu, shah@erev. ucf eddu Project Page: https::/joofiirreee 718 thub. WAAAALAA__WeeeeA
ABSTRACT
Bias in machine learning models can lead to unfair decision making, and while it has been well-studied in the image and text domains, it remains underexplored in action recognition. Action recognition models often suffer from background bias (i.e., inferring actions based on background cues) and foreground bias (i.e., relying subject appearance), which can be detrimental to real-life applications such as autonomous vehicles or assisted living monitoring. While prior approaches have mainly focused mitigating background bias using specialized augmentations, we thoroughly study both foreground and background bias. We propose ALBAR, a novel adversarial training method that mitigates foreground and background biases without requiring specialized knowledge of the bias attributes. Our framework applies adversarial cross cross-entropy loss to the sampled static clip (where all the frames are the same) and aims to make its class probabilities uniform using a proposed entropy maximization loss. Additionally, loss for regularization against the debiasing process. We evaluate our method on established background and foreground bias protocols, setting a new of-teoof and strongly improving combined debiasing performance by over 12% absolute on HMDB51. Furthermore, we identify an issue of background leakage in the existing UCF101 protocol for bias evaluation which provides shortcut to predict actions and does not provide an accurate measure of the debiasing capability of a model. We address this issue by proposing more fine-grained segmentation boundaries for the actor, where our method also outperforms existing approaches.
introduce gradient penalty
INTRODUCTION
In wide range computer vision tasks, models often rely on unintended and seemingly irrelevant patterns in the data, known as spurious correlations, as shortcuts to make predictions or decisions Geirhos et al. (2018) 2020 These correlations do not represent the true underlying relationship between the input features and the target output. As a result, models that exploit these spurious correlations may achieve high performance on the training and in-distribution test data, but fail to generalize well to wollw scenarios. A notable example of this is seen in video action recognition, where model will choose an action label by only considering the background instead of the subjects motion Ding al. (2022a); Zou al. 2023). For example, if subject is performing the action "Throwing Frisbee" while standing on a soccer field, a model will likely predict "Playing Soccer" instead. Here, the model is not using the motion information. instead using spatial information to make a biased decision. However, even if this background bias is mitigated, there may still be biases related the video foreground [i et al. 2023). Frisbee" indoors, but wearing soccer jersey, model may still erroneously predict "Playing Soccer". While the sources of foreground biases may include relatively harmless sources like a held object, they may also manifest in more harmful sources, such as person's 's physical appearance attributes like skin color, facial hair etc. Zhao et al. (2017); Stock & Cisse (2017); Buolamwini & Gebru (2018); Wilson al. (2019): Prabhu & Birhane (020);; Tong & Kagal (2020); Steed & Caliskan (2021): Gustafson et al. (2023). Such appearance- based decisions are highly undesirable in real-life applications of action recognition in security cameras, elderly monitor systems, autonomous cars
our example, even if the subject is "Throwing

---

Published as a conference paper at ICLR 2025
where an unbiased decision is crucial. Despite extensive studies background biases in action recognition, the area lacks comprehensive research on biases related to the foreground.
Adversarial learning has emerged promising method for debiasing neural network representations Beutel et al. (017); Elazar & Goldberg (2018); Zhang et al.] (2018); Wang et al. (2019). These techniques often require supplementary information, such as scene or object labels, involve training a separate critic model to predict the biased attribute. For example, scene/object classifiers Duan al. (2022) have been employed for mitigating scene bias in action recognition. The effectiveness of such methods is contingent upon the accuracy and reliability of the critic model, as they rely on its negative gradients for optimization. Since annotating video datasets with detailed attributes for biases requires enormous annotation efforts, it is not scalable. Li et al. [i et al. (2023) recently propose method to mitigate bias in action recognition using augmentations based on MixUp Zhang et al. (2017), where they first detect action-salient frames from videos and add such frames to other videos. Their idea is to reduce static bias by encouraging the model not to make decisions based on the mixed frames. Although this method works well in reducing the background bias, it does not significantly reduce the foreground bias and relies on an off-the-shelf salient frame detector model.
To address these challenges, we propose a novel adversarial learning technique that eliminates the need for attribute labels or pretrained attribute classifiers and provides an end-to-end training framework. Like Bahnng et al. (2020): [ao et al. (202I). we hypothesize that the bias issues in action recognition stem from an over-reliance on spatial information by the classifier. Prior works attempt to solve this by contrasting 3-dimensional (spatio-temporal) and 2-dimensional (spatial) representations using separate encoders. Instead, we break away from this formulation and design an adversarial framework based on single 3D encoder model. Specifically, we first sample any frame from within a video clip and repeat it to obtain a static clip. To mitigate the static bias, we introduce an adversarial loss through negative gradients to penalize the model from making action-class predictions based on static cues and propose entropy maximization loss to make its class predictions uniform across all classes. We also introduce a gradient penalty objective to regularize the debiasing process. We term our method ALBAR (meaning in Arabie: "Guard of All", Adversarial Learning approach to mitigate Biases in Action Recognition), for which a schematic diagram is shown in Fig.
We show state-of-the-arr performance on a comprehensive video action recognition bias evaluation protocol Li et al. (2023) based on popular benchmarks such as (00ttices & Zisserman (0177), UCF10l Soomro et al. (2012), and 1MDB51 Kuehne al. (2011). namely SCUBA (static cues in the background) and SCUFO (static cues in the foreground). SCUBA evaluates background bias by replacing the background of action clips, and SCUFO evaluates foreground bias by stacking a single frame from SCUBA to create clips with no motion. We also found a shortcut in the prior UCF101 bias protocol Li et al. 2023), which used bounding boxes to separate the foreground from the background, allowing background information surrounding the bounded subject to leak into the protocol. We propose a fix this version of the evaluation protocol that appropriately separates the foreground and background via segmentation masks.
The key contributions of this work can be outlined as follows:
We propose a novel adversarial learning- based method to mitigate biases in action recogni- tion, which provides simplified end-to-end training and does not require any labels/classifiers for bias-related attributes.
Our adversarial learning framework consists of loss and a gradient norm penalty, which are combined to strongly
saaetiie-grrdieen-baaeeee loss paired with
an matropy-maximizatie discourage the model from making predictions based on static cues.
Our method achieves strong sfate-of-the-ee art performance on established SCUBA/SCUFO background/foreground debiasing benchmarks: notably a strong 122% absolute increase in overall accuracy on the HMDB51 protocol.
Having identified shortcuts in the prior UCF101 bias protocols due to background informa- tion leakage, we resolve this by refining the test set with finer actor boundaries.
2
RELATED WORKS
General Bias Mitigation Many works have exposed various types biases in machine learning models Suresh & Guttag (2019) finding that not only do they pick up on biases within the training data, but
2

---

Published as a conference paper at ICLR 2025
they amplify them well Ntoutsi et al. (2020). This can be exceedingly harmful when the bias is related to demographic information, breaking fairness constraints Hardt et al. (2016). Additionally, training on a truly balanced dataset is virtually impossible Wang et al. (2019), and biases existing in a pretrained model tend to transfer into the downstream task [Salman et al. (2022). Therefore, is desirable seek solution like ours that mitigates biases the utility task level instead of the dataset level. Adversarial training Goodfellow et al. (2014); Xie et al. (2017); Zhang et al. (2018)
popular method that has proven effective in debiasing neural network representations Beutel et al. (017)) Elazar & Goldberg (2018); Zhang et al. (2018) 8 Wang et al. (2019). These techniques often necessitate the use of supplementary information, such as labels for scenes or objects present.
Alternatiely, these methods may involve the training of separate critic model to predict the biased
attribute, such as a gender classifier for gendered debiasing [Butee et al. 2017); Wang et al. (2019) or
scene/object classifiers Duan et al. (2022); /Zhai et al. (2023). They then rely on the negative gradients of the predictor network, making the effectiveness contingent upon the accuracy and reliability of the
critic model. In contrast, our adversarial method eliminates the need for specialized knowledge of bias attributes thorough labels or a separate predictor network. We simplify the debiasing process and reduce the computational overhead by utilizing the same model with different input.
Backkround/Foregroundd Bias in Action Recognition Most works exploring bias in action recogni- tion are focused mitigating the effect of the background representation biases, as models tend to use it over motion information to predict the action Yun et al. (2020); Choi et al. (2019); 9); Wang et al. (202I): Weinzaepfel & Rogez (202T); Ding et al. (2022a); Duan al. (2022); Byvshev et al. (2022). One approach to mitigate this involves emphasizing learning temporal information over spatial cues. This is often achieved through computationally expensive techniques such optical flow [Sun et al. (2018) : Sevilla-Lara et al. (2019) or by decoupling spatial and temporal representations Ding et al. (2022b): Zhang & Crandall (2022). However, these methods suffer from significant computational overhead, complex modeling techniques, or reliance on careful foreground/backgroumm mask annota- tions, limiting their practicality. One promising direction for learning quality temporal features is spatio-temporal contrastive learning Tao et al. (2020); SSchaapaa et al. (2023): Bahnn et al. (2020);
Bao et al. (021), but Wang et al. (2021) notice that in standard formulations, static information still tends to out-compete motion features. In contrast, static information, resulting in more strict utilization of temporal motion features. Li et al. (2023) reveals that many existing techniques which are resistant to background biases are still vulnerable to foreground biases. When the foreground video subject is a person, the foreground bias may be related to demographic information, which can cause inappropriate decision making. Taking this into account, our proposed adversarial leamning method accounts for all spatial information in both the video foreground and background.
enforce hard constraint the usefulness of
METHOD
The core of ALBAR, our proposed debiasing method, is to properly utilize an adversarial objective that discourages a model from learning to predict action classes based on spatial information alone. The additional proposed losses support this objective and help balance training for optimal performance. Each component of our method is shown in Figure T First, we notate the problem and baseline in Section [.1] Then, describe the core adversarial loss in Section (32 The additional supplementary entropy maximization and gradient penalty components are described in Section 3.3 and Section 3.4
respectively, with everything put together in Section 3.5
3.1 PROBLEM FORMULATION
A standard bias evaluation setup consists of two decoupled test sets: one that is independent and identically distributed (IID) from the training data, BDrDD and another one that is out-of-diistributio (OOD) DOoD. Formally, for video action recognition, we have video dataset ID = {xx1), . )
where x() is the ith video instance, (() is the associated one-hot action label, and N is the number of samples in the training dataset. A model is trained on D: then evaluated on both unseen test sets DILD and DooD- Debiasing methods attempt to learn robust and generalizable features that can maximize performance DooD without sacrificing performance on MDDD--
3

---

Published as a conference paper at ICLR 2025
(8) Clip X
Clip
Sample Single Frame
Gradient Penalty:
V
ll2
Lyp
shared weights
Encoder F.
LCE p y(())
Entropy Maximization:
(
(1)
P
yt
P
L
Uniform Distribation
LCE
LCE
Overall
L (1)
LCE + Wadv * ((0) + Went * C + wap
C( gp
Loss:
Figure 1: Given video clip x (i) we sample a random frame and stack it to create static clip ( Both clips are passed through encoder F to generate prediction vectors p and p!") The adversarial loss (Eq. is computed by taking the cross-entropy of the motion clip prediction p. (4) and subtracting the cross-entropy of the static clip prediction D This static prediction is encouraged to be uncertain by the entropy loss (Eq. 3). and the gradients related to the prediction (shown in red, Eq. 4 are encouraged to be lower for more stable training by the gradient penalty loss.
Baseline The conventional form supervised video action recognition employs standard empirical risk minimization (ERM) cross-entropy loss (Eq. to guide model predictions toward the ground truth label distribution y. The formulation as follows:
Nc Σ
CE
log !
(I)
where Nc is the total number of action classes in BrrD and Pa is the prediction vector. This objective is typically effective at maximizing performance on BID, but fails to properly generalize to a disjoint BooD Duan et al. (2022). Li et al. (2023) demonstrate that this loss allows for static information erroneously correlate with action labels, leading to staate-related biases being used in the final model predictions.
3.2 STATIC ADVERSARIAL LOSS
To solve this static bias problem, the correlation between static information and action label needs to be broken. Therefore, we propose an adversarial method to break this correlation by directly discouraging the model's '''s ability predict the action given single frame. Over the course of training, the model must learn to utilize primarily temporal information to achieve high performance, as discouraged from using spatial information. We hypothesize that reducing reliance on spatial information leads to more robust usage of temporal action information, allowing for reduced bias
and better generalization DOOD- Specifically, given input clip x(()) sample single frame at specified time f within x," stacking it times to create static clip, x! See Section 4.5 for discussion on [ frame selection. This new boring "clip" has zero motion. Both clips passed
4

---

Published as a conference paper at ICLR 2025
through backbone model F to generate prediction vectors Pi ( model is trained in such manner that the prediction p is still aligned to the ground-truth one-hot vector y()) but the cross-entropy between the prediction vector of the static clip D and y(i is maximized. This is accomplished by reversing the gradient of the cross-entropy loss of static-clip x Because these clips share high mutual information, differing only in motion patterns, the model should learn to utilize only the motion patterns to satisfy the loss conditions.
F(x{)) and D.
=(x;")) The
Formally, our adversarial loss objective can be expressed as follows:
ade = L ,y ()))
where (6) is the prediction vector of the static clip. This objective creates strong negative learning signal, discouraging the model from using static information in its classifications. Note that this method does not require the use of any additional labels achieve debiased representations.
3.3 ENTROPY MAXIMIZATION
Simply applying the two above objectives Eq. T & Eq. alone results in degraded performance (Tab. row b). We find that the model still learns the spatial correlations, but to satisfy the loss conditions, given static clip x(") it will predict incorrect class with high confidence. In order to combat this, we propose to directly add an entropy maximization loss to the static-clip prediction
(i) By encouraging the model to predict all classes with equal probability given clip vector Pi
with no motion, it prevents the label flipping trivial solution, forcing the model to rely on the clips with motion to make high-confidence predictions. The entropy loss is formulated as follows:
Nc Li
log
where (1) is the softmax prediction probability for class c. The summation computes the entropy of the predictions over all classes. The general formulation for entropy is negative, but since we want to maximize it, we negate the sum again, which gives us the loss function that we can minimize during training. The lower bound of this loss occurs when the prediction distribution is uniform, meaning that all classes have equal probabilities. This way, the encoder is trained to have higher uncertainty when not provided temporal motion information, ideally losing the ability to predict actions properly based on spatial information.
3.4 GRADIENT PENALTY
Even though the entropy loss helps prevent a trivial solution training collapse, we find that training is still unstable, experiencing major fluctuations in performance intermittent validation steps (see Appendix Fig. 6). The objective is for the encoder to learn better temporal representations, not directly react static inputs and drastically update weights. As such, need design loss that penalizes significant changes in the encoder's output when presented with static inputs. Taking inspiration from stabilizing GAN training Gulrajani et al. (2017). we add a gradient penalty loss. Instead of interpolating between samples and pushing the mean gradient norm towards 1, we simplify the formulation by directly minimizing the gradient norm with respect to the static clip x(O) This effectively acts as regularizer to stabilize the encoder by reducing sensitivity to static input, thereby causing the learning of more robust and generalizable motion features. The proposed gradient penalty loss is defined as follows:
(())
F(x
11) 1|2.
(4)
where ||-112 represents the & norm and V
represents the gradients of model FF w.r.t. static
clip input x In practice, this objective promotes smoothness and consistency in the adversarial gradients, leading more stable model convergence (Appendix Fig. 6
5

---

Published as a conference paper at ICLR 2025
3.5 COMBINED TRAINING OBJECTIVE
Putting everything together, our overall training objective is described by taking the average of all samples (i) E DIID computed using the following equation:
C
c + Wade - + Went C + Wap C op
(5)
where Wades Wents and Wgp are the relative weightage of each of the proposed loss functions.
4 EXPERIMENTS
4.1 DATASETS
Kinetics400 Carreira & Zisserman (2017) is large-scale action dataset that commonly used to pretrain models for a better initialization on various downstream tasks. However, the dataset has been shown to exhibit bias towards the background information. This can lead models to rely on background cues rather than focusing on the actual human actions.
UCF101 Soomro et al. (2012) a popular mid-sized action recognition dataset. Similar to Kinetics, it displays a high degree of background bias, i.e., the actions can be identified using the background.
HMDB51 Kuehne et al. (2011) smaller action recognition dataset. Unlike Kinetics400 and UCF101, HMDB51 has a large intra-class background variance, resulting in less reliance on static background information. Conversely, HMDB51 exhibits a relatively high degree of foreground bias, i.e., the actions can be identified using static foreground information.
ARAS Duan et al. (2022) a real-world test set based on Kinetics400. It is designed to be OOD for scene-debiasing evaluation by consisting of only examples of rare background scenes for each action.
SCUBA and SCUFO Li et al. (2023) are background and foreground bias evaluation benchmarks for action recognition based common benchmarks Kinetics400, UCF101, and HMDB51. SCUBA replaces the background action clips with alternate images from the following sources: the test set of Places365 /hou et al. (207)), generated by VOGAN-CLIP Crowson et al. (2022) , randomly VQGANN-CLPPCroooon
generated stripes following sinusoidal functions. SCUFO takes a frame from a SCUBA video and stacks it into motionless clip evaluate potential foreground bias.
4.2
IMPROVED UCF101 SCUBA/SCUFO PROTOCO1.
Li et al. 2023 proposed SCUBA and SCUFO datasets and metrics to evaluate both background and foreground bias in video action recognition models. These protocols require masks to extract the foreground from the original clips. However, the masks used for the UCF101 variation bounding boxes from the THUMOS-14 challenge Jiang et al.) (2014). Thus, surrounding background information is carried into the bias evaluation videos, as seen in Figure 2 This is not sufficient to evaluate performance on this dataset, as a classifier reliant on the background can still take advantage of this information to score high on the protocol. To mitigate this effect, we utilize a flexible video object segmentation model, SA..Track| Cheng et al. (2023): ): Yang et al. (202T); I Yang & Yang (2022);
Kirillov al. (2023): : Liu et al. (2023) to segment the actors (subjects) each video. The actors are initially grounded using the same THUMOS- 14 bounding boxes. Each testing video is manually checked for accurate segmentation. The same dataset creation protocol proposed by Li et al. (2023) is used create SCUBA and SCUFO variations with these new masks. The improved benchmark no longer includes background information, such as in Figure 2 (a), tightly bounding the human subject in Figure (b). Results in Table Sshow results on newly created benchmark. See Appendix Sec. for results on the existing benchmark.
4.3 IMPLEMENTATION DETAILS
For all experiments, we use a clip resolution of 224 x 224. We follow Li et al. (2023) and use Kinetics400 Carreira & Zisserman (2017) pretrained Swin-T al. (2022) with 32 frame clips at a skip rate of 2. We adopt the same common augmentations used in Li et al. (2023): random resized cropping and random horizontal flipping. Our chosen optimizer is Adam W Kingma & Ba (0014);

---

Published as a conference paper at ICLR 2025
(b)
Figure 2: Example clip from SE110--SCUBA-Siuusoi protocol clip, corresponding to a video from the class "Skiing". (a) shows the frames from previous protocol, where snow is visible in the background. Our improved protocol (b) uses tight segmentation masks to eliminate the background.
Table 1: Comparison results on III and OOD test sets of various debiasing methods on HMDB51. All experiments Swin-T pretrained using Kinetics-400. The light blue column highlights the contrasted accuracy.
OOD
Augmentation or Debiasing
IID
Avg SCUBA
Avg SCUFO
Confl- FG
Contra.
Acc.
None
73.92
43.93
20.46
36.58
27.84
MiXuPICTR'SS
1S
74.58
43.10
21.17
36.62
26.09
VideoMix
73.31
39.39
20.44
32.68
23.13
19
74.66
40.02
20.22
34.87
22.88
BECVPR'21
74.31
43.56
19,96
35.99
27.84
ActorCutMixevwur2
74.05
46.79
22.07
36.97
28.12
FAMEcvPR'2
73.79
51.40
26.92
39.61
29.66
StillMixiccv223
74.82
51.81
13.39
47.38
40.28
Ours
72.81
53.53
1.50 (92.7%
48.13 131.4%
53.22
191.2%
Ours w/ StillMix aug.
74.31
54.24
1.35
47.07
53.68
Loshchilov & Hutter (017)) with default parameters 31 0.9, 32 0.999, and weight decay of 0.01.
follow the linear scaling rule Goyal et al. (017)) with a base learning rate le-4 corresponding to batch size of 64. For training, utilize a linear warmup of 5 epochs and cosine learning rate scheduler. Further details may found Appendix Section
4.4 BACKGROUUNDFFOREGGOINDD BIAS EVALUATION
On existing background/foreegruund bias benchmarks, follow the usual protocol of reporting the average Top-1 accuracy across 3 runs. We first perform IID evaluation using the original test sets, then evaluate OOD performance using the SCUBA and SCUFO datasets. On the SCUBA protocol, higher accuracy shows a lower static background bias, yet on the SCUFA protocol, a lower accuracy shows lower static foreground bias. There also an additional conflicting foreground version (Confl- FG) that adds random foreground from one SCUBA video to another in the sinusoidal set. We also report contrasted accuracy Li et al. (2023) (Contra. Acc.), where a video prediction is counted correct if and only if the model is correct video. This single score is representative of overall bias reduction performance both the foreground and the background, and is highlighted in light blue in each table due its importance. Further explanation this protocol can be found in Appendix Section A and Figure 4 We compare against previous debiasing action recognition techniques and augmentations Mixup Zhang et al. (2017).
SCUBA video and is incorrect on the corresponding SCUFO
7

---

Published as a conference paper at ICLR 2025
Table 2: Results on IID and OOD of various debiasing methods on UCFI01 using our proposed updated protocol. The light blue column highlights the contrasted accuracy. All experiments use Swin-T pretrained using Kinetics-400.
OOD
Augmentation or Debiasing
IID
Avg SCUBA
Avg SCUFO
Confl-y FG
Contra.
Acc.
None
95.51
18.78
1.50
49.36
17.53
StillMix 1CCV'23
96.14
24.68
0.36
55.03
24.40
Ours
94.98
26.25
139.8% 31.17
0.14 (,0.7%
54.02 19.4%
26.23
149.6%
Ours w/ StillMix aug.
94.71
0.16
58.84
31.14
VideoMix Yun et al. 2020), SDN Choi et al. (2019), BE Wang et al. (2021 ActorCutMix Ding et al. (2022a), and StillMix Li et al. (2023).
The results on the HMDB51 variant are shown in Table 1 Results for the updated UCF101 version can be found Table 2 with results on the old protocol in Appendix Section C Remarkably, ALBAR improves upon the previous contrasted accuracy by over 12% for a total of 53.22% on the HMDB51 protocol. We see that our static adversarial method strongly reduces harmful biases by properly learning robust motion features for video action recognition, without requiring bias attribute specific knowledge. Additionally, as ALBAR employs a combination of training loss objectives and no major augmentations, we can easily combine the StillMix Li et al. (2023) augmentation with our method during training. This pushes the performance further beyond what either method can achieve alone, achieving a contrasted accuracy of 53.68%.
ARAS Rare Scene Evaluation Performance on ARAS is evaluated by computing top-1 accuracy using Kinetics400 trained classifier (the same used in SCUBA/SCUFO protocol). Table 6 in Appendix Section C includes results on both benchmarks. ALBAR is able to properly mitigate real-world scene-related biases, not getting fooled by even extreme cases rare backgrounds.
(a) Ground Truth Action: "throw"
(b) Ground Truth Action: "kick"
(c) Baseline Prediction: "swing baseball"
(d) Baseline Prediction: "fencing"
(e) ALBAR Prediction: "throw"
(f) ALBBAR Prediction: "kick"
Figure 3: Qualitative examples from the HMDB51 test set showing the baseline model choosing incorrect action label due to spatial context. Our method correctly chooses the action label in each. These pixel-level attributions plotted using integrated gradients Sundararajan et al. (017)
8

---

Published as a conference paper at ICLR 2025
Qualitative Results [ two examples of misclassifications of the baseline model that were correctly interpreted by our debiased model. We plot the pixel-level attributions for the predicted class using integrated gradients Sundararajan et al. (2017) in Figure Interestingly, looking the baseball_the t_trotttwwwwr example, we see that both the baseline (Fig 3cp and our method (Fig (ee have similar attribution maps, appropriately looking at the video foreground. The baseline model still chooses the incorrect action class, indicating potential foreground bias, likely using the baseball uniform. In Figure (bb the camera angle appears similar instances of fencing, but the correct action is actually "kick". In each of these examples, the biased baseline likely utilizes spurious features in the classification, but ALBAR easily overcomes them.
4.5 ABLATIONS AND ANALYSIS
Training loss components: Table abblates each individual loss component during debiasing training. Each piece is crucial for achieving maximum performance. The adversarial loss strongly contributes to the foreground debiasing performance by imposing a strong penalty for identifying the correct action utilizing static information. Alone (row b), it causes the model to degenerate into trivial solution where the model learns the inappropriate static correlations, then intentionally chooses the incorrect class. The entropy maximization objective is useful by itself (row c) in reducing reliance on static information, but is not strong of an objective as Lade- It combines with the adversarial loss (row e) to prevent the trivial solution and further improve debiasing performance, but training remains unstable (see Appendix Figure The gradient penalty does not have a large effect on its own (row d), but has positive effect when combined with the other losses, likely because the penalty prevents the static clip gradients from offsetting proper learning from the motion clip gradients. Once all three objectives are utilized simultancously (row h), we see a strong gain in performance. Each component complements each other well, leading our method to achieve SOTA contrasted accuracy on the SCUBA/SCUFO benchmarks. Additional ablations can be found in Appendix Section C
Table Ablation to determine the effectiveness of each component loss objective.
OOD
LCF Ladv
Lent
gg
IID
Avg SCUBA
Avg SCUFO
Confi-+ FG
Contra
Acc.
(a)
X
x
73.92
43.93
20.46
36.58
27.84
(b)
72.61
42.29
0.00
32.77
42,29
(c)
X
71.57
46.02
2.97
38.40
44.30
(d)
72.09
41.13
14.69
35.16
29.24
(c)
72.22
46.81
0.49
41.13
46.72
(
71.70
45.92
0.00
38.52
45,92
(g)
73.86
49.14
9.74
42.89
41.11
(h)
/
72.81
53.53
1.50
48.13
53.22
Static frame selection: Table ablates the frame sampling strategy used to chose for the static clip. Specifically, we evaluate choosing the first, middle, last, and random frame positions. Choosing the first and last frames sharply decreased performance, likely due to scene changes or irrelevant information before and after actions. Contrasting the action learning from irrelevant information is trivial and would not learning anything useful. Random frame selection ensures variety in static objectives, leading to strong results. Notably, middle frame selection improved performance over random selection. This makes sense, since the middle frame is likely to contain the full background and actor in the middle of performing an action, making a hard negative sample for the adversarial learning process. These findings highlight the importance of choosing static frame where the action currently being performed, as should include both the foreground and background information. While using sophisticated method to detect actors/backgrounds in frames and choosing static frames based on the presence of both would likely achieve more optimal performance, this adds too much inductive bias and computation,
leave this up future work.

---

Published as a conference paper at ICLR 2025
Table 4: Ablation evaluate static frame sampling strategy.
OOD
Frame Sampling
IID
Strategy
Avg SCUBA
Avg SCUFO
Confl-+ FG
Contra.
Acc.
Random
73.20
53.22
0.42
49.84
53.02
First
72.75
50.92
0.18
45.59
50.91
Middle
72.81
53.53
1.50
48.13
53.22
Last
72.68
49.49
0.40
42.42
49.40
Our results thus far demonstrate the improved capabilities of our method in the constrained problem of trimmed action recognition. Here, we show that debiasing video encoder leads to performance gains various downstream video understanding tasks. Specifically, we evaluate on weakly supervised anomaly detection and untrimmed temporal action localization, both of which take in long videos and localize the timestamps where anomalies/specific actions occur. In each of these tasks, feature quality and feature discriminability are crucial to performance. Having an unbiased encoder allows for localized representations to depend on motion rather than the background or biased foreground. For the weakly supervised anomaly detection task, report results the UCF Crime (Sultani et al. (2018) dataset frame-level Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC), which is the standard evaluation metric for this task. We use hyperparameters for this evaluation, only swapping the feature sets used to those extracted from baseline model and a model trained using our framework on HMDB51. For the temporal action detection task, we report results on the THUMOS 14 Jiang et al. (2014) dataset. Evaluation is given as mean average precision (mAP). We use SOTA model TriDet Shi al.) (2023) with standard hyperparameters, again only swapping the feature sets used in similar fashion to the UCF Crime protocol. More details may be found Appendix Section Table 5 highhights results, demonstrating the benefit of our debiasing method across downstream video understanding tasks.
Downstream task evaluation:
of the current SOTA methods MGFN Chen et al. (2023) with unchanged
Table 5: Evaluation of our debiased model across downstream anomaly detection temporal action localization (THUMOS14) tasks.
and
Action Recognition HMDB51 (OOD)
Anomaly Detection UCF _cimm
Temporal Action Localization THUM0S14
Method
Baseline
27.84
82.39
54.89
Ours
53.22
84.91
55.20
Limitations Our evaluation focuses on established background and foreground bias protocols, which may not capture all possible real-world scenarios and biases. Expanding the evaluation include more diverse and realistic settings, including those with harmful demographic biases, would provide a more comprehensive assessment of the practical effectiveness approach makes an assumption that static information should be useful. Future work could explore architectures that still eliminate spatial biases while still utilizing the information appropriately.
our method. Additionally, our
CONCLUSION
We propose ALBAR, a novel adversarial training framework for efficient background and fore- ground debiasing of video action recognition models. The framework climinates the need for direct knowledge of biased attributes such as an additional critic model, instead leveraging the negative cross-entropy loss of a clip without motion passed through the same model as the adversarial com- ponent. To ensure optimal training, we incorporate static clip entropy maximization and gradient penalty objectives. We thoroughly validate the performance of our approach across comprehensive suite of bias evaluation protocols, demonstrating its effectiveness and generalization across multiple datasets. Moreover, ALBAR can be seamlessly combined with existing debiasing augmentations to achieve performance that significantly surpasses the current state-of- state-of-the-art. is our hope that our work contributes to the development of fair, unbiased, and trustworthy video understanding models.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENTS
This work was supported in part by the National Science Foundation (NSF) and Center for Smart Streetscapes (CS3) under NSF Cooperative Agreement No. EEC-2133516.
REFERENCES
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528-539. PMLR, 2020.
Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13349-13358, 2021.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv: 2017.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transpurency, pp. 77-91. PMLR, 2018
Petr Byvshev, Pascal Mettes, and Yu Xiao. Are 3d convolutional networks inherently biased towards appearance? Computer Vision and Image Understanding, 220: 103437, 2022.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi, and Yik-Chung Wu. Mgfn: Magnitude-contrasive glance-and-focur network for weakly-supervvsed video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 387-395, 2023,
Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06555 2023.
Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can't dance in the mall? learning to mitigate scene bias in action recognition. Advances in Neural Information Processing Systems, 32, 2019.
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pp. 185. Springer, 2022.
Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong. Motion-aware contrastive video representation learning via foreground- background merging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 9716-9726. 2022a.
Shuangrui Ding, Rui Qian, and Hongkai Xiong. Dual contrastive learning for spatio-temporal representation. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5649-5658, 2022b.
Haodong Duan, Yue Zhao, Kai Chen, Yuanjun Xiong, and Dahua Lin. Mitigating representation bias in action recognition: Algorithms and benchmarks. In European Conference on Computer Vision, pp. 557-575. Springer, 2022.
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. arXiv preprint arXiv:1808.006 066400 2018
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. taggenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 12231, 2018.

---

Published as a conference paper at ICLR 2025
Robert Geirhos, Homm-eerikk Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence,
2020.
lan Goodfellow, Jean Pouget-Abadicg Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:/706.0000 7766002677 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans, Advances in neural information processing systems, 30, 2017.
Laura Gustafson, Chloe Rolland, Nikhila Ravi, Quentin Duval, Aaron Adcock, Cheng- Yang Fu, Melissa Hall, and Candace Ross. Facet: Fairness in computer vision evaluation benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20380,20382, 2023.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Action recognition with a large number of classes. http::/crcvv.cff. ucf. edu/ THUMOS147 2014.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 11X1v:1412.6980, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
Hildegard Kuchne, Hucihan Jhuang, Estibaliz Garrote, Tomaso Poggio. and Thomas Serre. Hmdb: large video database for human motion recognition. In 2011 International conference on computer vision, pp.
IEEE, 2011.
Haoxin Li, Yuan Liu, Hanwang Zhang, and Boyang Li. Mitigating and evaluating static bias of action representations in the background and the foreground. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19911-19923, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre- trainininng for open-set object detection. arXiv preprint 2303:23033054990 2023.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings the pp. 3202-3211, 1, 2022.
the IEEE/CVF conference computer vision and pattern recognition,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 11Xiv:1711.05101, 2017.
N Ntoutsi, P Fafalios, U Gadiraju, V Iosifidis, W Nejdl, ME Vidal, S Ruggieri, F Turini, S Papadopou- los, E Krasanakis, et al. Bias in data-driven ai systems- an introductory survey. arxiv 2020. arXiv preprint arXiv:2001.099 099662 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perii peghomerformance deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Vinay Uday Prabhu and Abeba Birhane. Large datasets: A pyrrhic win for computer vision. arXiv preprint arXiv:2006. 220220006.6623.. 16923, 3, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /119999111.,11111 2019.
Hadi Salman, Saachi Jain, Andrew lyas, Logan Engstrom, Eric Wong, and Aleksander Madry. When does bias transfer in transfer learning? arXiv preprint arXiv:2207.028-2 2022.
Madeline Schiappa, Yogesh $ Rawat, and Mubarak Shah. Self-supervised learning for videos: survey, ACM Computing Surveys, 55(13s):1-37, 2023.
Laura Sevilla-Lara Yiyi Liao, Fatma Güney, Varun Jampani, Andreas Geiger, and Michael J Black. On the integration of optical flow and action recognition. In Pattern Recognition: 40th German Conference, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings 40, pp. 281-297. Springer, 2019.
Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: Temporal action detection with relative boundary modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18857-18866, 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12 2222204022 2012.
Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human- like biases. In Proceedings of the 2021 ACM conference on fairness, accountability, and iransparency, pp. 701-713, 2021.
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Explanations, bias detection, adversarial examples and model criticism. arXiv preprint arXiv:1711.11444 2017
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6479-6488, 2018.
Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1390-1399, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Harini Suresh and John V Guttag. A framework for understanding unintended consequences of machine learning. arXiv preprint arXiv: 10002, 2(8), 2019.
Li Tao, Xueting Wang, and Toshihiko Yamasaki. Self-supervised video representation learning using inter-intra contrastive framework. In Proceedings of the 28th ACM International Conference on Multimedia, pp.
2020.
Schrasing Tong and Lalana Kagal. Investigating bias in image classification using model explanations. arXiv preprint 2rXiv:2012.05463. 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Jinpeng Wang, Yuting Gao, Ke Li, Yiqi Lin, Andy Ma, Hao Cheng, Pai Peng, Feiyue Huang. Rongrong Ji, and Xing Sun. Removing the background by adding the background: Towards background robust self-supervised video representation learning. In Proceedings of the IEFF/VV Conference Computer Vision and Pattern Recognition, pp. 11804-1181, 2021.
Tianlu Wang. Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets not enough: Estimating and mitigating gender bias in deep image representations. Proceedings of the IEEF/CVF international conference computer vision, pp. 5310-5319, 2019.
13

---

Published as a conference paper at ICLR 2025
Philippe Weinzaepfel and Grégory Rogez. Mimetics: Towards understanding human actions out of context. International Journal of Computer Vision, 129(5):1655-1699, 2021.
Benjamin Wilson, Judy Hoffmman, and Jamie Morgenstern. Predictive inequity in object detection.
arXiv preprint arXiv:1902.11000 2019.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning, Advances in neural information processing systems, 30, 2017.
Zongxin Yang and Yi Yang. Decoupling features in hicrarchical propagation for video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020.
Yuanhao Zhai, Ziyi Liu, Zhenyu Wu, Yi Wu, Chunluan Zhou, David Doermann, Junsong Yuan, and Gang Hua. Soar: Scene-debiasing open-set action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2224-10254, 2023.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335- 340, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXi:: 2017.
Zehua Zhang and David Crandall. Hicrarchically decoupled opatial-temporal contrast for self- supervised video representation learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3235-3245. 2022.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint 17X1::1077.04557 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions patlern analysis and machine intelligence, 452-4444411444 2017.
Yuliang Zou, Jinwoo Choi, Qitong Wang, and Jia- Bin Huang. Learning representational invariances for data-efficient action recognition. Computer Vision and Image Understanding, 227: 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDIX OVERVIEW
Section Dataset details
Section B
Implemenntation/ccmmpiie details
Section
Additional experiment details
A
DATASET DETAILS
Kinetics400 Carreira Zisserman (2017) contains approximately 300,000 video clips sourced from YouTube, covering 400 human action classes. It has single dedicated train/val/test split. In this work, we train on the train split and evaluate IID on the test split.
UCF101 (Soomro et al. (2012) comprises of 13,320 video clips across 101 action classes and has three train/test splits available. Following Li et al. (002)., we utilize only the first (split 1) train/test split for all training and evaluation in this work.
HMDB51 Kuehne et al. (2011) consists of 6,849 video clips covering 51 human activity classes and has three potential train/test splits, much like UCF101. Again, only use the first (split 1) train/test split for all training and evaluation in this work.
ARAS et al. (2022) is only test set, all of the 1,038 rare-scene action videos utilized to compute Top-1 accuracy.
SCUBA and SCUFO Li et al. (023) also test sets. Each dataset variation contains three variations of different background types to ensure solid generalization to OOD data. All of the data the related SCUBA and SCUFO splits is combined as described in Main Paper Section 4.4 to compute the contrasted accuracy. Figure includes additional bias evaluation dataset examples.
used compute Top-1 accuracy, and performance
SCUBA
Foreground: original
Background: replaced
Motion: original
Goal: high accuracy
SCUFO
Foreground: original
Background: replaced
Motion: none
Goal: low accuracy
ConfIGG
Foreground:
original + distractor
Background: roplaced
Motion: originul
Goal: high accuracy
Figure 4: Example SCUBA/SCUFO/ConIFG frames from HMDB51. SCUBA evaluates background bias while SCUFO samples single frame, evaluating foreground bias. ConfIFG adds static distractor foreground to SCUBA video, evaluating both bias types simultaneously. Contrasted accuracy requires predicting the correct label on SCUBA video AND predicting the incorrect action on its paired SCUFO video. More information can be found in StillMix Li et al. (2023).
Our proposed fix to the UCF101 protocol is applied across all three background types, namely Places365 images, VQGAN-Clip images, and random sinusoidal images. Further visual examples of the improved protocol are included in Figure 5 Note that the "Fencing" example, we additionally segment the opposing fencer for complete video.
15

---

Published as a conference paper at ICLR 2025
(a)
(d)
Figure 5: Example clips from SEFIO1-SCUBBA-Pacces33 and UCFIOI-SCUBA-VOOS protocols. (a) shows an example video from the class "Fencing" from the previous protocol. Our improved protocol (b) uses tight segmentation masks eliminate background information. Likewise, (c) shows an example clip from the class "Golf Swing". and (d) shows the improved segmented version.
B IMPLEMENTATION DETAILS
This section is an addition the details listed in Main Paper Section 4.3 A standard validation set does not exist for HDMB51 and UCF101. We randomly sample 20% of the respective training sets to use for validation, labelling them with an identical process as in 4.2 The Py Paszke et al. (2019) library is utilized for all experiments. All experiments cluster with access to V100 and A100 GPUs of various memory configurations up to 80GB. On an 80GB GPU, batch size of 8 clips is used, and train for up to 50 epochs. On HMDB51, this may take = hours with validation every epoch. We follow [i et al. (2023) and evaluate Top-1 accuracy using a single clip sampled from the center of each video. The only augmentations test time resize to short side 256 and a center crop to 224 x 224.
performed
local computing
C ADDITIONAL EXPERIMENT DETAILS
Expanded tables: Table shows expanded (non-averaged) SCUBA/SCUFO results for HMDB51, and Table 8 shows them for the original UCF101 protocol. In the original UCF10L protocol, the test set contains a significant amount extra background information, leading to skewed results, especially the SCUBA background bias protocol. The background is mostly replaced, but good portion of the actual video background remains, inflating scores. Our method does not utilize this well, seemingly performing worse result.
As referenced in Main Paper Section 4.4 results on the Kinetics400 and ARAS versions can be found here, in Table 6 Similar outcomes to the other experiments seen here.
16

---

Published as a conference paper at ICLR 2025
Table 6: Results IID and OOD test of various debiasing methods Kinetics400, All
experiments use Swin-'T pretrained using Kinetics-400.
OOD
Augmentation Debiasing
IID
Avg SCUBA
Avg SCUFO
Contra.
ARAST
Acc.
None
68.13
42.97
20.26
57.57
25.78
StillMix 1CCV
67.27
45.83
10.72
59.21
36.60
Ours
67.58
44.75
14.1%
0.11 199.5%
59.79
13.9%
44.74
173.6% %
Ours w/ StillMix aug.
68.10
46.52
0.77
60.17
46.07
Table 7: Quantitative results of various debiasing methods on the HMDB51 Kuehne et al. (201))
dataset. The light blue column highlights the contrasted aceuracy. Bold and underline indicate best and second best results, respectively.
Augmentation Debiasing
HMDB51 HMDB51-SCUBA. (f)
HMDB51-SCUFO (U)
Contra Acc. (
HMDB51
Places365
VOGAN-CLIP
Sinusoid
Places365
VQGAN-CLIP Sinusoid
None
73.92
47.61
42.77
41.41
20.68
17.90
22.80
27.84
Mixup
74.58
46.70
42.49
40.12
21.25
18.47
23.78
26.09
VidcoMix
73.31
41.33
38.18
38.67
19.64
18.82
22.85
23.13
SDN
74.66
41.96
40.82
37.29
19.99
19.62
21.06
22.88
BE
74.31
47.36
42.94
40.39
20.91
17.55
21.41
21.41
ActorCutMix
74.05
50.13
46.51
43.73
22.16
20.26
23.80
28.12
FAME
73.79
54.71
53.67
45.81
27.10
27.26
26.40
29.66
StillMix
74.82
53.27
52.43
49.73
13.39
12.66
14.13
40.28
Ours
73.20
54.73
54.80
50.12
0.04
1.17
0.04
53.02
Ours w/ StillMix aug.
74.31
55.59
56.95
50.20
1.45
1.52
1.09
56.68
Table 8: Quantitative results of various debiasing methods on the UCF101 Soomro al. (2012) dataset, using the existing protocol from et al. (202)), which contains background information. The light blue column highlights the contrasted accuracy. Bold and underline indicate best and second best results, respectively.
Augmentation or Debiasing
UCFI01-SCUBA (1)
UCFI0I-SCUPO (L)
Contra Acc. (
UCFI01
Placea365
VQGAN-CLIP Sinusoid Places365
VQGAN-CLIP Simusoid
No
96.21
37.63
34.37
54.94
3,48
3.02
10.82
36.82
Mixup
96.17
39.82
40.89
57.79
2,88
3.28
11.62
40.46
VideoMix
96.00
28.59
37.36
58.26
7.81
11.40
20.60
29.37
SDN
95.76
34.78
32.56
50.40
2.21
1.42
5.30
36.42
BE
96.06
39.76
36.16
56.01
3.55
2.93
10.15
38.62
ActorCutMix
95.87
51.02
55.28
69.53
8.00
8.43
19.32
46.87
FAME
95.81
40.62
44.56
37.54
5,74
6.50
6,84
35.14
StillMix
96.02
55.22
53.68
65.75
2.40
2.16
5.76
54.90
Ours
94.98
46.04
42.36
55.70
0.68
0.59
0.34
47.88
Ours w/ StillMix aug.
94.79
57.19
54.78
60.91
0.13
0.31
0.22
57.58
Downstream task evaluation details: For the downstream task evaluation presented Main Paper Table we utilize official implementations from MGFN Chen et al. (2023 for anomaly detection and Trilet Shi al. (2023 for temporal action localization. The only difference between the baseline and our method is the extracted feature set input to the task models. The baseline uses
Swin-T model finetuned on HMDB51 with the standard cross-entropy objective. Our ALBAR trained
17

---

Published as a conference paper at ICLR 2025
encoder corresponds to the model shown in our best reported results on HMDB51 finetuning. The exact provided hyperparameters are utilized and unchanged from the original implementations.
Method Generalization Example- Image Classification Debiasing: Our method was built based on an observation of a semi-unique property of video - the fact that a single frame from along the temporal dimension contains all the same 2D information information necessary to classify the end result: an action taking place across time. At the core of this is the idea that we have paired inputs: one containing all necessary information for classification, and one containing mostly the same information, but lacks crucial information to complete the task. Given this unique problem setup in another setting, our methodology should apply. As such, we have put together quick experiment to evaluate our method in an alternate, albeit related, domain: image classification. Similar to action recognition, the background bias problem is well-known and well studied. In the Waterbirds (Sagaaa et al. (2019) dataset (based on CUB-200- (201T), D, images are built by taking segmented images of specific bird types and placing them on specific background types (land birds, waterbirds vs. land-based, water-based backgrounds), causing models to learn the spurious correlation between the bird type and background type. We observe similar phenomenon to our problem setup, where instead of having the temporal dimension to reduce, we can reduce information in the spatial dimensions by removing the foreground. Here, our pairing becomes the original image (containing the foreground bird for classification) and the original background image (with no bird, so it should be useless for bird classification). Acquiring these pairs in a non-synthetic setup takes more effort than our video-based setup, but our core method should nonetheless still apply. The results in Table | indicate that there is merit to our method outside of video understanding. Our ALBAR method improves both minority classes and gorst-group accuracy without spending time to optimize hyperparameters. While this is not robust analysis, we believe that in setting up these two scenarios (video debiasing, image classification debiasing), we demonstrate the broader applicability of our method across domains, contingent upon having the unique paired input setup.
3D clip, but it does not contain the
Wah et al.
Note: Since the standard Waterbirds dataset does not separately contain references to the background images used, we had to create a split (using the public code provided by the original authors), modifying it to additionally save the background images to create our pairing.
Table 9: Generalization experiment using Waterbirds Sagawa et al. (2019). Per-class accuracy (%) evaluation is provided, with worst-group accuracy commonly used as an evaluation metric. WoW- Waterbirds on Water, LoL = Landbirds on Land, etc.
Method
WoW (majority)
WoL (minority)
LoL (majority)
LoW (minority)
Baseline (R18)
92.68
50.00
99.16
76.45
Ours (R18)
92.99
59.50
98.67
78.00
Effect of gradient penalty: Figure 6 highlights the efficacy of our additional gradient penalty objective in stabilizing training and improving performance. prevents the model from taking large steps in different directions as the static adversarial and temporal cross-entropy objectives fight back and forth, leading to overall smoother learning and improved overall performance.
Strength of ad adversarial component Wads: : The adversarial loss weight (Wadv) has the most significant impact on training efficacy. Our empirical findings suggest that the optimal value for wadv is 1, which equates to the cross-entropy weight of the motion clip. Suboptimal values lead to two contrasting issues: (1) Wadv < 1 insufficiently mitigates static biases, while (2) Wade > 1 dominates the learning process, overshadowing the crucial information contained within the motion clips. Striking the perfect balance is key to achieving effective training and ensuring the model learns from both the adversarial loss and the motion clips in complementary manner.
Strength of entropy maximization Went! in model stability. Inadequate values lead to two distinct failure modes: (1) an insufficiently low weight allows the classifier to misclassify static inputs by assigning incorrect predictions, while (2) excessively high coefficient impairs the classifier's capacity capture and learn temporal dependencies effectively. Much like the adversarial loss, balancing the entropy maximization objective's strength is essential for the model achieve stable and accurate performance.
The entropy objective coefficient plays a critical role
18

---

Published as a conference paper at ICLR 2025
With Gradient Penalty
Without Gradient Penalty
50
HMDB51 Contra. Acc. (%)
45
35
30
20
10
20
30
Epoch
Figure 6: Stability of performance during training with and without the proposed gradient penalty objective.
Table 10: Ablation with adversarial loss Lado weightage.
OOD
Wade
IID
Avg SCUBA
Avg SCUFO
Confl- FG
Contra.
Acc.
0
73.92
43.93
20.46
36.58
27.84
0.5
73.79
48.65
3.31
43.28
46.56
73.20
53.20
0.42
49.84
53.02
2
67.84
44.52
0.99
40.86
44.45
4
51.50
27.76
5.86
24.02
27.76
Strength of gradient penalty wipp: Similar to the other objectives, stable training relies on finding an optimal loss weight Wap- Ineffective weights lead to two failure cases: (1) a low wyp renders the penalty insufficient for acting proper regularizer, while (2) higher coefficient can overshadow the primary training objective and cause the gradients to become oversmoothed, resulting in reduced performance on DIDD-
Table 11: Ablation with entropy loss Lens weight. Table 12: Ablation with gradient loss Lyp weight.
OOD
OOD
Went
IID
Avg SCUBA
Avg SCUFO
Confi-, FG
Contra. Acc.
Wgp
IID
Avg SCUBA
Avg SCUFO
Confl-, FG
Contra. Acc.
73.92
43.93
20.46
36.58
27.84
0
73.92
43.93
20.46
36.58
27.84
72.61
51.71
0.00
44.57
51.71
5
73.59
49.77
0.03
44.53
49.77
73.20
53.20
49.84
53.02
10
73.20
53.20
0.42
49.84
53.02
74.12
49.87
1.86
43.01
49.26
20
71.76
50.55
3.03
43.55
48.84
19


Reviews:
[{"review_id": "poHDoYD15A", "reviewer": "Reviewer_5f7A", "summary": "This paper proposes a novel adversarial learning-based method to mitigate biases in action recognition, which provides simplified end-to-end training and does not require any labels/classifiers for bias-related attributes.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper is well-written and well-organized.\n+ Good performance on the popular action recognition datasets.", "weaknesses": "This paper introduces adversarial learning into action recognition, which is a relatively novel idea, but I have some concerns as follows.\n\n- Are there more visual examples that can depict biases in action recognition?\n- Whether the method proposed in this paper can be used for skeleton-based action recognition task？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KO1DSikc2l", "reviewer": "Reviewer_EAVn", "summary": "The paper proposes a method to improve generalization capability of an activity recognition method in temporally-segmented video datasets.  The method focuses on an adversarial approach to removing biases from static elements of the scene.  The paper takes a random frame from the video and creates a \"static\" video by repeating the frame the same length as the original clip, then using this static clip as the adversary.  The paper combines this adversarial approach with other reasonable loss terms.  It demonstrates state of the art accuracy on the problem.", "questions": "- The mechanism for adversarial learning in the paper is very specific to action recognition.  How can this mechanism be generalized to be of broader interest to the ICLR community?\n\n- The method works by sampling any frame from the video and repeating it for a static clip.  Why any frame?  Aren't certain frames better or worse than others for the stated goal?  (The pre-segmented nature of the datasets in question create itself an algorithmic bias.  In the stated real world application deployments no such pretemporal segmentation is avavilable, and hence brings into question the feasability of the method in practice.)  But, more concretely to the task, why is there no analysis whatsoever on the impact of this frame selection?  Even for a subset of one dataset, it would have been interesting to understand the breadth of potential with different static clips.\n\n- Wouldn't it be clearer to concretely specify that the $\\vby$ action label notation is a one-hot vector?  It is one-hot, right?\n\n- At line 218, shouldn't \"maximized\" be \"minimized\"?  At least, something in that sentence does not match up: \"p(t) is still matched to gt distribution y, but the similar ....maximized\"  Maximizing the \"similarity\" (a loose term here) is minimizing the ce loss.\n\n- How are there reasonable IID results in Table 3 for the case that Ladv is not used ---> when it is not used, there is no actual gradient to guide the model to do any recognition?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of bias mitigation in ML is seriously important.  The paper proposes a concrete approach to mitigating clear bias problems in action recognition.\n\n- The entropy maximization term makes sense for avoiding the static cues.\n\n- The paper uses practical mechanisms to overcome the challenges of training the models.\n\n- The evaluation is performed on a recent OOD benchmark for bias analysis, and performs well relative to other methods.\n\n- The writing is mostly clear and concrete.", "weaknesses": "- The paper is is somewhat \"small\" in that although it describes what seems to be a new method and its evaluation on reasonable benchmarks, there is little discussion around key elements of the method, their limitations, their rationale, etc.  (see questions)\nEDIT: The review discussion thus far, and the other reviewers seems to have identified similar concerns.  Discussion around this point suggests there is some generality to visual problems, but some concern about the narrowness of the contribution remains.  Most of the comparative papers are in CV conferences.  I raised my scored, but this point should be discussed among the AC pair/triplet however ICLR is doing it this round.\n\n- Does the \"static clip\" way of approaching video bias mitigation, have broader utility in video understanding or beyond?\n\n- Certain interesting experiments are omitted, even though the writing suggests they may be useful.  For example, L233 reads \"A naive application of Eq. 2 results in degraded performance.\"  What exactly is a naive application?  Furthermore, this implies that the two parts of Eq. 2 is individually interesting; in particular, does the right hand side do anything?  The ablation study does not include these two parts separately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9VizzsY3C", "reviewer": "Reviewer_aFRh", "summary": "This paper presents ALBAR, an adversarial learning method aimed at reducing biases in action recognition models. The study focuses on addressing background and foreground biases, which can impact the performance of applications such as autonomous vehicles and assisted living monitoring. ALBAR utilizes an adversarial training technique to minimize the model's dependence on static background cues, encouraging the use of dynamic motion information for action classification.", "questions": "Although the results demonstrate that the model outperforms state-of-the-art methods, the technical contribution is incremental and purely based on previously introduced techniques.\n\nHow is the fairness of the comparative experiments ensured in this paper, and how are the results of the comparative methods obtained?\n\nThe ALBAR performs well on specific bias evaluation protocols, but its generalization capabilities to new types of biases or different domain tasks have not been fully validated.\n\nAlthough the paper proposes a simplified end-to-end training framework, adversarial training often involves additional computational costs. The paper does not discuss the computational efficiency and scalability of the ALBAR method in detail.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "S1: This paper resents state-of-the-art results on established SCUBA/SCUFO background/foreground debiasing benchmarks, and formulation is technically sound.\n\nS2: Several ablations show the contribution of each component.", "weaknesses": "W1: Although the results demonstrate that the model outperforms state-of-the-art methods, the technical contribution is incremental and purely based on previously introduced techniques.\n\nW2: How is the fairness of the comparative experiments ensured in this paper, and how are the results of the comparative methods obtained?\n\nW3: The ALBAR performs well on specific bias evaluation protocols, but its generalization capabilities to new types of biases or different domain tasks have not been fully validated.\n\nW4: Although the paper proposes a simplified end-to-end training framework, adversarial training often involves additional computational costs. The paper does not discuss the computational efficiency and scalability of the ALBAR method in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)