A unique M-pattern for micro-expression spotting in long videos
Authors: Jinxuan Wang
Shiting Xu
Tong Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Micro-expression spotting
Optical flow
Facial alignment
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Micro-expression spotting (MES) is challenging since the small magnitude of micro-expression (ME) makes them susceptible to global movements like head rotation. However, the unique movement pattern and inherent characteristics of ME allow them to be distinguished from other movements. Existing MES methods based on fixed reference frame degrade optical flow accuracy and are overly dependent on facial alignment. In this paper, we propose a skip-$k$-frame block-wise main directional mean optical flow (MDMO) feature for MES based on unfixed reference frame. By employing skip-$k$-frame strategy, we substantiate the existence of a distinct and exclusive movement pattern in ME, called M-pattern due to its feature curve resembling the letter `M'. Based on M-pattern and characteristics of ME, we then provide a novel spotting rules to precisely locate ME intervals. Block-wise MDMO feature is capable of removing global movements without compromising complete ME movements in the early feature extraction stage. Besides, A novel pixelmatch-based facial alignment algorithm with dynamic update of reference frame is proposed to better align facial images and reduce jitter between frames. Experimental results on CAS(ME)$^2$, SAMM-LV and CASME II validate the proposed methods are superior to the state-of-the-art methods.

Published as a conference paper at ICLR 2024
A UNIQUE M-PATTERN FOR MICRO-EXPRESSION SPOT- TING IN LONG VIDEOS
Jinxuan , Shiting Xu Tong Zhang 12.
South China University of Technology & Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human Pazhoo Lab, Guangzhou, China {2 {202221043813 csxst @mmail. scut. edutte.dducccmm en, tony@scut. edu.s on
ABSTRACT
Micro-expression spotting (MES) is challenging since the small magnitude of micro-expression (ME) makes them susceptible to global movements like head ro- tation. However, the unique movement pattern and inherent characteristics of ME allow them to be distinguished from other movements. Existing MES methods based on fixed reference frame degrade optical flow accuracy and are overly de- pendent on facial alignment. In this paper, we propose skip-k-frame block-wise main directional mean optical flow (MDMO) feature for MES based on unfixed reference frame. By employing skip-k-frame strategy, we substantiate the exis- tence of distinct and exclusive movement pattern in ME, called M-pattern due to feature curve resembling the letter 'M'. Based on M-pattern and characteristics of ME, we then provide novel spotting rules to precisely locate ME intervals. Block-wise MDMO feature is capable of removing global movements without compromising complete ME movements in the early feature extraction stage. Be- sides, A novel pixelmatch- based facial alignment algorithm with dynamic update of reference frame is proposed better align facial images and reduce jitter be- tween frames. Experimental results on CAS(ME)? SAMM-LV and CASME II validate the proposed methods are superior the sfte-of-the-arr methods.
INTRODUCTION
Spontancous micro-expression (ME) is unintentionally leaked when people attempt to conceal or restrain their expressions in an intensely emotional situation (Li et al. (023a)). These expressions are of low intensity and usually last less than 0.5 second (Ben et al.) 2022), making them difficult to be detected. Unlike macro-expression (MaE), once ME appears, can reflect the most genuine emotion, which is why ME plays an indispensable role in the fields of psychology. business negoti- ation, and criminal investigation (Xu et al. 2022). Micro-expression spotting (MES) refers to locate segments (i.e., the onset and offset frame) of micro expressions in long video. Global movements, including head movements, zooming in and out are almost inevitable and numerous over a extended time span. The brief and subtle nature of ME makes it almost imperceptible to the naked eyc. Under such circumstances, ME is easily eclipsed by prolonged and large global movements.
With the advancement of computer vision (Zhang et al. 2024 Xu et al. 2023 Zhang al. 202.3
MES research has evolved mainly into traditional and deep learning methods. The latter primarily draws inspiration from the fields of temporal action localization (Yang et al. (020). object detec- tion TTan et al.] 2020) and visual recognition (Wang et al., 2021 Liu et al.) 2024). Considering the enormous variety of expression scales, feature pyramid network (FPN) widely employed in MES (He et al. 2022] Yu et al., (222]) However. deep learning methods are limited in various aspects, e.g., inadequate ME samples, disturbance from global movements and limitations of unimodal data. Recent developments in self-supervision and generative modeling can mitigate these flaws. Tradi- tional methods resort to a prior knowledge of the ME field (Ekman & Friesen, 2019, Bhushan (015) and mainly include three steps: (1) facial alignment, (2) feature extraction, and (3) spotting. Facial aligmment is essential (Yap et al. 2022) as it aims to remove head movements. Mainstream optical
"Corresponding Author

---

rules Published as a conference paper at ICLR 2024
flow (OF) based alignment methods (Yuhong 2021 Zhao et al.) 2022) align all frames with fixed reference frame (RF). However, due to massive global movements, OF cannot characterize subtle displacements properly. Dominate feature extraction approaches similarly based on fixed RF, which compute the OF between each frame and the RF to describe the changes between them (Yuhong) 2021 ZZang al.. 2020). These methods overly rely on the performance of facial alignment, head posture and facial appearance differences between subsequent frames and RF are too significant to satisty OF assumptions. Another problem is that these methods prefer to extract feature within the regions of interest(Rols) to capture subtle movements selectively and reduce computation time, causing the inability to observe global movements. Thus many false micro movements are wrongly detected as ME, Spotting is designed to locate ME intervals. Zhang et al. (2020) utilize the property that ME moves opposite directions during the occurrence and restoration phases to reduce false positives (FP). Li et al. (223b)) exploit the localization character of ME for spotting. But they all fail to fully utilize the characteristics of ME to distinguish them from other micro movements.
Based on the limitations above, we adopt an unfixed RF strategy for both facial alignment and feature extraction. We first propose a pixelmatch-based facial alignment algorithm with dynamic update of reference frame improve alignment performance and reduce jitter. In order to minimize the disturbance of global movements, we propose a f--amme block-wise main directional mean optical flow (MDMO) feature based on unfixed RF. Block-wise MDMO is capable of simultane- ously extracting features and eliminating global movements. By using a skip-k-frame strategy, we substantiate a unique movement pattern of ME, called M-pattem. M-pattern capable of precisely describing the complete ME movement from occurrence to restoration, and it is targeted and accu- rate for spotting ME intervals even without facial alignment. Then we provide a novel spotting based on M-pattern and three characteristics of ME to spot ME intervals.
2 RELATED WORKS
Facial alignment is widely used for its significance in eliminating global movements. Yan et al.
(2144) utilize Local Weighted Mean(LWM) transformation to align the facial landmarks of each frame with manually selected RF, but this cause severe facial deformation. Yap et al. (2020)
apply the online toolbox OpenFace (Baltrusaitis et al. (018)) for facial aligmment. Zhao et al. 2022)
and He et al. (2020) utilize OF as displacement measure, and align each frame with the RF (nor-
mally the first frame) according to the displacement in the nose region. The above methods all adopt a fixed RF strategy, with better alignment for nearby frames but worse for distant ones.
Common feature descriptors include MDMO (Liu et al. 2016). LBP-TOP (Tran et al. 2011)) HOOF (Verburg& & Menkovski 2019), etc. Main directional mean optical flow (MDMO) (Liu et al. 2016 is one of the most representative features in MES. It divides the OF field into 8 equally sized and disjoint bins and averages the OF vectors in main direction, hence obtains a more robust and rep- resentative feature to describe movement. Zhang et al. (2020) divide the OF field into 4 unequally sized and disjoint bins, which is more line with the directions of ME movement. ME reveals itself in specific facial regions (Ma et al. 2019), so it is redundant to analyze the whole face. Existing methods mainly focus on features around eyebrows and mouth. Yuhong (021) extracts 14 RoIs to analyze MDMO in each Rol. Zhang et al. 2020) extract feature in 12 Rols to generate 3D matrix to analyze spatio-temporal information. However, these localized extraction methods prevents them from observing the global movement, leading to many of them being recognized as ME.
Two solutions are available for addressing the disadvantages of fixed RF strategy: clip-based and unfixed-RF-based. Clip-based methods divide a whole video into multiple clips and perform MES on each clip individually. Waang et al.] (202T) propose a clip proposal network to filter negative samples. Yang al. (2021) and Tran et al. (2021) employ a sliding window strategy scan across the video sequence. Li et al. (2023b) extract temporal feature over 300 ms (based on ME duration), and they discover a S-pattern in feature curve when ME occurs. However, the S-pattern can only describe half of the ME movement from onset to apex. Lifixed-RFFbaseed methods implement fixed interval to calculate the difference between two frames. Han et al. (018) and Li et al. (2018) compute the difference between current frame fi and the average feature frame (i.e., the average feature of fi-k and f(+) to spot apex frame. Liong et al. (2021) extract OF feature between f: and fitk instead of f, and RF. Due to the short time span between fi and fitk. it ensures the OF can correctly depict the movement between them. However, they ignore the fact that during the ME restoration phase, computing OF in this way leads to additional peak in feature curve.
2

---

Published as a conference paper at ICLR 2024
Facial Alignment
Feature Extraction
Spotting
Pixelmatch-haseel alignment
Blocks and Rols localization
M-pattern detection
Reference frame updating
Skip-k-frame block-wise MDMO feature extraction
Micro-expressions spotting
Figure 1: Overall framework. The proposed methods consist of three parts focusing on MES and global movements climination. First, we introduce pixelmatch-basee facial alignment algorithm with dynamic update of reference frame to align facial images. Second, we propose skip-k-frame block-wise MDMO to extract features with subtle changes and eliminate global movements simul- tancously. Under skip-k-framee strategy, we verify unique movement pattern of ME, called M- pattern. Third, based on M-pattern and ME characteristics, we provide novel ME spotting rules.
PROPOSED METHOD
3.1 FACIAL ALIGNMENT
Aligning all frames with fixed RF in long videos is recommended and the OF-based method is significantly influenced by head displacement. Thus we propose a pixelmatch-base facial align- ment algorithm with dynamic update reference frame. Pixclmatch(Mapbox 2022) Python package used for fast pixel-level image comparison. It is based on the RGB difference and re- turns the number of changed pixels between two images, denoted by m. As shown in Fig. 2, the positions of the matching and cutting box are defined on the basis of center coordinate c(æ, y)- The sizes of boxes are defined on the basis of two measures of the human face: horizon- tal distance between the inner eye corners and vertical distance between the nasal spine point and the line connecting the inner eye corners(Li et al.) 2018). The nose is chosen the match- ing box due to its stability and rigidity. The algorithm consists of alignment and reference frame update (pseudo-code and details in Ap- pendix A.1). Alignment is the process of ad- justing the center coordinate until m between the current matching box and the reference matching box is minimized. Mmin is set to reduce the jitter between images, which means no adjustment will be made if the mismatch pixels is less than M Miin- Since the head posture changes over time, necessary to adjust the RF dynamically. But we must ensure that the updated RF is well-aligned to prevent subsequent frames from aligning with a misaligned frame. If m is equal to 0 or greater than Mmazs the RF updated as the current frame. = 0 means perfectly aligned. m > Mmmar implies significant change in head pose or facial appearance, making subsequent frames difficult to align if the RF is fixed. If 0 < m < M the RF will not be updated temporarily, until the frame with more satisfactory alignment appears.
cutting
bing
Figure 2: Cutting box and matching box.
3.2 M-PATTERN
With short time span, skip-k-frame strategy (N + 1)/2, N is the average ME duration) ensures that the two frames involved in computation satisfy the assumptions of OF (minor displace- ment and constant apparent brightness) even without facial alignment. We denote the jth Rol of the äth frame as fL and the OF magnitude between f and f as p.. The calculation details will be discussed in Section 3.3. As shown in Fig. 3, we define complete ME movement from fi to fi+22. with fi+k being the apex frame. Theoretically. two neutral states fa and fi+2k are identical with lower magnitude (i.e., lower muscle deformation intensity). The intensity starts increase mono- tonically as the expression develop from a neutral state (f.) to a peak state (f++)). Then decreases the expression restores from apex back to another neutral state(/++20 Under skim-k-frame ap-
3

---

Published as a conference paper at ICLR 2024
proach, the feature curve of such ME exhibits particular pattern, called M- M-pattem, as shown in Fig. The details ) are described below.
M-pattern
intensity
ME
frame
frame
(+
(+
i+ 1+ 3k/2 1+ 2k
1+ 3k
The muscle deformation inten- Figure 4: Skip-k-frame OF magnitude sity curve of a complete ME.
Figure 3:
of a complete ME.
Point denotes the OF magnitude between fi-k and fi. The expression is in the initial stage of occurrence and the facial state is relatively stable before f.. Therefore p nearly 0.
Point @ is the first peak in feature curve. begins to increase as the ME develops from fi to litk. where fü in neutral appearance, and fi+* is in the highest intensity of the facial motion.
Point ② is the local minimum in this feature curve. The onset-apex and apex-offset phases of ME are actually opposite and symmetrical movements. Therefore, p begins fall until it reaches local minimum value in the vicinity of fi+32/2- For example, consider a frowning expression, where the halfway-furrowen appearance in lowering eyebrows phase closely resembles the halfway-furrower appearance in raising eyebrows phase.
Point is the second peak feature curve. p starts to increase again because one of the frames is progressing towards the apex(fi+k/2 - fi=i), while the other is recovering towards neutral (fi4k//2 - f422.). ). In fact, and are maxm-ssymettric as the ME curve in Fig. is also y-axis-symmetric.
Point is almost analogous to The facial appearance has fully restored to a neutral state, thus little difference can be found between fi+2k and fet3k.
Eventually, we get a trajectory shaped like 'M', with a span from fi to fi+3k- But notice that although get feature curve with two waves, it is obtained by single ME, and the final predicted interval is the start of the first wave to the peak of the second wave, not the entire 'M'.
CASME II Yan et al. 2014), CAS(ME) Qu et al. 2018) and SAMM-LV Yap et al. 2020 applied to the validation of the M-pattern. Appendix A.2 illustrates the feature curves of ME on three datasets. We can find that they all display the same pattern similar to letter 'M', Thus utilize the M-pattern for MES.
3.3 SPOTTING RULES
We aim to exploit the unique properties of M- M-pattern to differentiate ME from other movements. Thus, we propose spotting rules based on M-pattern and ME characteristixx(Suulley, Localization, Transience). It is divided into 6 steps specifically.
Step 1. Peak detection and interval generation. These are applied to locate the peaks with potential rapid facial movements. The threshold is calculated as:
(1) where Crean and Cstd are the average and standard deviation of all local maxima in a video. p is parameter to determine the threshold level. The set CT comprises local maxima greater than T. As shown in Fig. 5, For each peak in CT we extend the left and right of p, to the relatively flat part of feature curve(or extend to trough like the middle of 'M', if there is an adjacent wave next to p:), to generate an interval w, = [w, w.).
T Cmean + Catd:
Step 2. Potential ME spotting based on M-pattern. According to the inference of M-pattern, if two intervals (waves) are close enough to each other, i.c., satisfying Eq. (2):
w k/2 < w..
(2)
4

---

Published as a conference paper at ICLR 2024
can combine them into one interval from the start of the first wave to the peak of the second wave. This is the combination rule when the ME is at average length. Practically, the duration of some MEs is too short/long, and the duration of apex-offset phase of most MEs is longer than the onset-apex phase (Yan al. 2013). In these cases the combination rules will be slightly different. For example, if ME is from f. to fi+k, the predicted interval of this ME is from the start to the end of the first wave since the second wave of 'M' is actually a replica of the first one. We use 4,,6 to denote the distance from to b. The specific combination rules can be determined by the lengths of two adjacent waves:
G
Ci
arg
framg
w/ - 3ps fas
W
wf (wf.1)
wf-1
Figure 5: Symbolic markers of feature curve.
[wrg, mr(),
if luequuf S k: if k < hogguu S 2k, twi..ps S K,
[wsf. w/ 20
52k,
[wr, Pet1),
if k: < m if bfg > 2k.
uu S 2hi, buy .p > K
[ws], Pi
Based on M-pattern, a complete ME consists of two waves, thus Wi+2 is the next be analyzed after wi, not W/+1- In fact, not all MEs have restoration phase. There is a circumstance when the expression reaches the peak state and then continues to remain in that state until the next expression occurs. The latter half of the curve in Fig. 3 will remain at the peak level. The feature curve of this type of ME has only one wave, instead of two waves like the M-pattern. Thus, w; does not satisfy Eq. (2), we determine whether w; is single wave according to Eq. (4). ( + k < w and w w((+1)
wf.
(4)
In this case, it is easy deduce the predicted interval is the ascending part of the wave, i.e., [w] psl- And w is the next to be analyzed, not Wi+2-
Step 3. First selection based on subtle and localized nature of ME. E = {e/li = 0, 1.... denotes the set of ME intervals of jth Rol after step 2, and of is the peak value of each ME interval. If ef is generated by a double wave, of is the larger one two peaks. Subtlety means that the intensity of ME is much gentler than MaE and other movements. So we set maximum threshold Tmaz to exclude intervals with large c/. Existing spotting rules treat each Rol individually, but the expression are the result of coordinated movements across multiple Rols. For a MaE consisting of multiple Rols, some of which have intensity greater than Tiaa and some less than Tmaz. it is impossible to exclude such MaE if each Rol is treated separately. Thus, we reduce FP at the expression level, not at the RoI level. Specifically, intervals with IoU greater than 0.5 belong to the same movement. We introduce ddeg to describe the overall facial muscle deformation degree for an expression, and ddeg! is the deformation degree for jth Rol, ith interval:
(5) where aeg' is the average 3 fps frames preceding the start frame of e.. ddeg is the sum of ddeg belonging the same movement. If ddeg > Tadeg or the largest ddeg; in the same movement is greater than Tmaxs all of belonging to this movement will be excluded simultancously. The second feature is localization, which is reflected in the fact that ME involves fewer Rols. If a movement engages more than half of the RoIs, then all el constitute this movement are excluded.
ddeg c// //nngg
Step 4. Second selection based on global movements. The movement of nose tip region is approx- imated as global movement because of its rigidity. Steps 1-2 are applied to the nose tip region to detect possible global movements, and ei will be excluded if it intersects with global movements.
Step 5. Merging. The merging involves fusing intersecting intervals, i.e., intervals belonging to the same expression, from different Rols. Suppose there are two intervals a [a", af] and = [ be] from different RoIs with loU iouab. If iouab IoU mougergs the two intervals are merged into one [a, b). where
max(a", b,)" 0.5 x (max((a b²) minna", b))), b = min(a" 5b) + 0.5 x (max (a", bt) min(a" 6
(6)
5

---

Published as a conference paper at ICLR 2024
This is extension the left and right with the intersecting intervals as the center. It ensures the length of merged interval is almost equal to the average length of intervals involved in merging.
Step 6. Last selection based transient nature of ME. Transience refers to the short duration of the ME (<0.5s) (Yan et al. 2013), which allows us to exclude intervals with duration longer than 0.5s.
Hence, have obtained all the predicted expression intervals that both satisfy the M- M-pattern and all the characteristics of ME.
3.4 FEATURE EXTRACTION
Since global movements are unidirectional, i.e., no restoration phase, the feature curves will be single-wave, the same the feature curve of ME without restoration phase. Thus, propose the block-wise MDMO to extract features within Rols and eliminate global movements in the early feature extraction stage.
Specifically, as shown in Fig. 6(a), we divide the facial area into 9 blocks to analyze the global movement and extract 15 Rols, including eye regions detect ME move- ments and blink. Original MDMO et al., 2016) divides the OF field into 8 disjoint bins on average in Fig. 7. However. if the directionn of movement falls between two ad- jacent bins, MDMO will only average the vectors within the bin with maximum count(i.e., the main direction), resulting in deviation in both magnitude and orientation of MDMO. For this reason, we divide the OF field into 8 intersecting bins as illustrated Fig. 6(c). Each pixel will belong to two different bins, and directions that were between two adja- cent bins now fall into one bin. Each bin in conjunction with its two intersecting bins constitutes direction group. For each block R* = 0, I..., 8), we classify all the OF vectors into 8 bins according to their ori- entation and compute MDMO. Denote the pixel counts distribution in R as B 1t). 0h orgmaa((b), b/.....b)) the main direction. The MDMO of Rd is compute as:
(b,, b.,,
(a) Blocks and RoIs divi- sion.
(b) Mask.
Σ
u (p),),
(7)
0x.7
Bl max
4
LAS
ue (p) BLnxx
where |
max(b) bq,... U$), and B is the set of OF vectors in main di- and = provide very clear descrip-
5112
3
625
rection. B == -BO, B¹ BB,.,.,B B8)
(60, 01 ..... (.,....000) 08) tion the global movement. Similarly, MD- MOs of 15 RoIs, denoted as w', are computed following Eq. (7). Subsequently, is com- mon zonnghag et al. 2020) remove the global movement using the displacement in nose tip region. However, The OF vectors distribution in Fig. illustrates that this turns out to be limited to the translational head movements. For circumstances such as head rotation, brightness change, zooming in and out, etc., it can only lead to worse results. Therefore, we define five states (or- der, disorder, rotation, adjustment, blink) to categorize each frame according to B and θ with the purpose of eliminating global movements without disrupting the complete ME movements.
(c) intersecting bins of optical flow field.
Figure 6: Block-wise MDMO.
Order. Order means all 0 point at the same direction group and 8 elements in B* for each block should be ordered and consistent with 8² i.e., 3 bins in the direction group should contain all the OF vectors in R and the remaining 5 bins should be close to 0. Normally, frames with translation and other minimal head movements will be classified as order. The MDMO of the nose tip region is basically in line with the global movement. Therefore still subtract u nose to remove the global

---

Published as a conference paper at ICLR 2024
2
4
0
5
(a) Translation.
(b) Rotation.
(c) Image blurring.
Figure 7: 8 disjoint bins in orig- inal MDMO.
Figure Optical flow images of different circumstances.
movement, The final OF feature for 1 is computed
i. u u nose
(8)
Disorder. Disorder means that the OF vectors are chaotic and random. Using alone sufficient for determining whether frame is disorder. Sometimes all 0* points in the same direction group, but the OF vectors within each block are very disorganized. For R². min(B") > diisorder: then R is disorder. In other words, each bin (direction) has significant portion of vectors belonging to it. Brightness change and image blurring are most likely to cause disorder. this case, MDMO of neither the blocks nor the nose tip can accurately represent their movement, but the MDMO of Rols happen to have moderate magnitude to generate fake ME movement. Therefore, for all Rols, we set their final OF the average OF in video:
2 i=k+1
(9)
N
Rotation. After facial alignment, the head rotation is almost centered on the nose tip. Excluding center block 4, the main directions of the other 8 blocks are mostly different. If we visualize 8 MDMOs with arrows, they should overall conform to a clockwise or counterclockwise circle or oval(See Fig. 8(b)). Head rotation is most likely to generate fake ME movement as its magnitude very close ME. So the final OF feature is the same as Eq. (9).
Adjustment. When 9 8h do not all belong to the same direction group and the global movement is
not rotation or disorder, simply following Eq. (8) doesn't 't really help. We use th represent the local movement of Rt and u² should minus the MDMO of the block Rol j belongs to, not uoose to eliminate the global movement.
= - u. (fR E R
(10)
Blink. Neutral blinking sometimes causes a slight vibration in eyebrows with other RoIs beyond the eyebrows being not affected. It is easy to detect blink using the left and right eye region since there are only upward and downward movements of the upper eyelid. If blink happens under an ordered
state, the final OF of Rols in eyebrow area should follow Eq. (9), and other Rols should follow Eq. (8).
If frames within a ME span are classified as one of the other four states except for 'order', the M-pattern probably will be compromised since their OF vectors are altered. If there is a chance that fi is during ME movement, the feature of this frame should be calculated according to Eq. (8), regardless of the state it is categorized into. The subtle nature of ME is reflected in the fact that muscle changes are largely confined to the Rols, with noticeable changes occurring in other facial region. This is why pixels within 15 Rols are not involved in the calculation of the MDMO for 9 blocks, i.e., only the black area in Fig. 6(b) participates in the calculation of प. z² is used to describe the local movement of k-th block. Excluding pixels within the RoIs does not affect the final result since t is averaged, but including pixels within the Rols when a ME occurs makes /h closer to describing the movement of the Rol rather than the local movement of the block. Therefore, if the magnitude of | wt l, (f? E R!)) is larger than Taiff, which means possible ME movement may occur, then di must be computed according to Eq. (8) no matter what state is classified as.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
Table I: Comparison with the ofate-of-the-ae arr methods.
CAS(ME)
SAMM-LV
CASME II
Methods
ME
MaE
All
ME
MaE
All
ME
MDMO He et al. 2020
0.0082
0.1196
0.0376
0.0364
0.0629
0.0445
SP-FD et al.
20200
0.0547
0.2131
0.1403
0.1331
0.0725
0.0999
OF-FD Yuhong 1
0.1965
0.3782
0.3436
0.2162
0.4149
0.3638
SOFTNee Liong el al.
2021
0.1173
0.2410
0.2022
0.1520
0.2169
0.1881
TSMSNet Xue al. 2021)
0.1969
0.2395
0.2275
0.0878
0.2515
0.2466
(BPN(Leng et al. 2022)
0.1590
0.3357
0.3117
0.1689
0.3349
0.2908
AUW-GCN( Yin et (023))
0.1538
0.4235
0.3834
0.1984
0.4293
0.3728
S-pattern( Li et al.
2023b)
0.4700
Ours
0.2614
0.5061
0.4558
0.2866
0.3724
0.3419
0.8571
We conduct the experiments on CASME II Yan et al. 2014), CAS(ME) Qu et al. 2018 and SAMM-LV (Yap al., 2020) and follow the performance metries proposed in MESNet (Wang al.
2021I. Appendix A.3 and A.4 introduce the details of datasets, metrics and experiment settings.
Table 1 reports our spotting results three datasets together with the taato-of-thh-artSSOT meth- ods. Our proposed MES algorithm outperforms the others on all three datasets. Each video in CASME II contains only one zero ME and little head movement. We only perform step 1, 2, and 5 for MES in CASME II. The Fl-score of 0,8571 and the recall of 0.8823 sufficiently demon- strate that the feature curve of ME conforms the M-pattern under skip-k-frame calculation and M-pattern is more suitable for describing the ME movement in comparison to S S-pattern (Li et al. 20236). Except for MaE spotting results SAMM--V, the significant improvement in Fl-score on CAS(ME) and SAMM-LV is mainly attributed to the global movements elimination by bise MDMO in the feature extraction phase and the robustness and distinctiveness of M-pattern. The overall results indicate that the M-pattern applies to both macro and micro-expression spotting in long videos and our spotting rules are capable of distinguishing ME/ MaE from other movements. The main reason why our MaE spotting results in SAMM-LV is roughly 5% lower than the SOTA is that we have not extracted the Rols below the eyes as proposed by He et al Yuhong 20211. and these Rols contain considerable portion of MaEs related to blinking and eye movement.
We find that FP in ME spotting results are composed of three types of movements: habitual move- weth the highest proportion, such eyebrow raising, pursing lips and pouting), MaE move- ments, and ME movements that do not satisfy Eq. (11). ). Habitual movements are almost identical with labeled ME in both magnitude and direction of movement. Thus, it is impossible to distin- guish such habitual movements from ME using the unimodal data only. No global movements are detected in our results, which again demonstrates the superiority of our proposed methods in elimi- nating and distinguishing global movements. The detailed spotting results of our proposed methods are displayed in Appendix A.5.
Parameter Analysis. We focus on analyzing the effect of Taifs on spotting results. Tays is the difference between the Rol MDMO पं and block MDMO य,. It is designed to not compromise the possible ME movement when removing global movements based on the state of each frame. Fig. illustrates the impact of varying Tasfs from 0 to 0.6 on the spotting results. Taifs = means all the frames are treated as the order state. Hence no ME is compromised. In general, When Tass > 0, we observe a large decrease in the number of FP for all spotting results in comparison to Taff = 0 and the Fl scores have also improved. This is a solid evidence that block-wise MDMO can effectively eliminate global movements in the feature extraction phase. The number of TP (True Positives) at Tasfs 0 is almost equal to the number at Taiff > O, which proves that by setting Taff,, we successfully preserve the possible micro-movementts while eliminating the global movements. Tasyy is set to 0.4, 0.6, 0.2, 0.3, in the same order as the legend in Fig. 9. Setting larger Tass might be able to eliminate more global movements, since most expressions have a magnitude exceeds 1.0. However, the larger Tais/ / is, the more likely it is to disrupt the complete expression movements, especially for MaE, where the large magnitude of movement can easily cause the frame to be categorized disorder state.
8

---

Published as a conference paper at ICLR 2024
CAS(ME) ME
CAS(ME) MaE
SAMMLV ME
SAMMLV MaE
Specinay Aegs
Sp Alo
5paingg AABY
(a) F1-score
(b) FP
(c) TP
(a) F1-score
(b) FP
(c) Recall
Figure 9: Spotting results for different Tais
Figure 10: Progressive spotting results across four spotting steps.
Efficacy of spotting rules. Fig. 10 illustrates the results from spotting step 2 to 6. Step 1 is de- signed to generate waves hence no spotting results are expected. We present the results after step 5(merging) for better analysis and visualization. As can be seen in Fig. 10 (a) and (b), each step capable of eliminating FP according to a specific rule, leading to gradual increase in F1-score. Step 3 and 6, spotting based the characteristics of ME, contributes the most. It eliminates considerable amount of FP that do not meet the ME characteristics, but at the same time, it also leaves TP with small magnitude out. The reason why step 6 is executed after merging is because we want to perform from the perspective ment by step 6 is more significant in ME than in MaE, because ME in CAS(ME)² and SAMM-LV have a distinct duration boundary, whereas MaE do not. The recall of SAMMLV MaE spotting step is the lowest in Fig. 10 (c) and the overall performance is as superior the other three. The first reason has been explained above. The second reason is probably because larger k(130) are less suitable for the proposed spotting algorithm than smaller (7, 18, 37, 34).
complete merged expression interval. The improve-
Table 2: Ablation experiments.
Since facial expressions are the result of coordinated movements across multi- ple Rols, the proposed spotting rules are based on the expression level. We conduct experiments
CAS(MB)2
SAMM-LV
ME
MaE
ME
MaE
bwMDMO
0.261
0.506
0.287
0.372
bw bwMDMO, Rol-level
0.233
0.486
0.278
0.363
Rol-level with the same spotting rules and the results are shown in Table 2. Expression-level is more superior than Rol-level in all spotting results. For ME, spotting at expression-leve can elim- inate intervals with small magnitude that are likely to be generated by MaE or other movements. However, these interval are inclined to be kept if we spot at the Rol-
bWMMMO, w/o step3,
0.159
0.455
0.204
0.358
bwMDMO, w/o step4
0.224
0.498
0.279
0.367
bwMDMO, w/o FA
0.200
0.430
0.231
0.353
bwMDMO, w/o [FA, step4]
0.190
0.449
0.229
0.348
Fixed RF, w/o FA
(Yuhong 2021
0.117
0.291
0.153
0.279
bwMDMO: Block-wise MDMO. Rol-level: spotting at Rol-level. FA: Pixelmatch-base facial alignment.
level, for their magnitude are closely similar ME.
Efficacy of facial alignment. We conduct experiments on the raw video to verify the performance of the proposed facial alignment algorithm. The results reflect the effectiveness of our alignment method. We also verify the advantage of the skip-k-frame strategy over the fixed RF strategy in the absence of facial alignment. Due to short time span, the Skip-k-frame strategy has already reduced global movements to a great extent at the very beginning, whereas the fixed RF strategy heavily relies on facial alignment to climinate global movements.
5 CONCLUSION
In this paper, we propose skip-k-frand block-wise MDMO feature for MES. Under skip-k-frame strategy, we substantiate a novel unique M-pattern for describing complete ME/ MaE movements. The spotting rules are designed based on M-pattern and ME traits. Block-wise MDMO can simulta- neously extract features within Rols and eliminate global movements. Dynamic updating of the RF method enables the proposed pixelmatch-bas based facial alignment algorithm to reduce jitter between frames and enhance alignment performance. Experimental results demonstrate the efficacy of the proposed methods.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was funded in part by the National Key Research and Development Program of China under number 2019YTA0706200, in part by the National Natural Science Foundation of China grant under number 62222603, 62076102, and 92267203, in part by the STI2030-Major Projects grant from the Ministry of Science and Technology of the People's Republic of China under num- ber 20212D0000700, in part by the Key-Area Research and Development Program of Guangdong Province under number 202380303030001 and in part by the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (20192T08X214).
REFERENCES
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference Automatic Face &amp; Gesture Recognition (FG 2018), May 2018. doi: 2..1099ff.2018.0000 8.00019. URL http:
doi.org/10. /10 1109 fg. 2018 00019
Xianye Ben, Yi Ren, Junping Zhang, Su-Jing Wang, Kidiyo Kpalma, Weixiao Meng, and Yong-Jin Liu. Video-based facial micro-expression analysis: A survey of datasets, features and algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5826-5846, 2022. doi:
10. 20097TPAMM222213306777
Braj Bhushan.
Study of Facial Micro-expressions in Psychology, pp. 265-286. Jan
2015. doi:
978 81 - 322 10110077978-88332221994-----
URL http://dx. doi. http://ct..dd..rg/111...... wtt..d....rg111110000711..1111..
Paul Ekman and Wallace V. Friesen. Facial action coding system, Jan 2019. URL http://dx, doi org/10r..1.... 1037/277334-00
Yiheng Han, Bingjun Li, Yu-Kun Lai, and Yong-Jin Liu. Cfd: A collaborative feature difference method for spontaneous micro-expression spotting. In 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 1942-1946, 2018. doi: 10. 109/ICIPP200 1065.
Xiaosong He, Xiao Wu, Jun Peng, Qingxia Li, Xinkai Ma, and Yuanmin He. Bidirectional cross- scale feature fusion for long video micro-expression 3d spotting network. In 2022 IEEE 21 International Conference Cognitive Informatics & Cognitive Computing (ICCI*CC), pp. 110- 115, 2022. doi: 10.1 (0.110///CICCC50042200000 10101155.
Ying He, Su-Jing Wang, Jingting Li, and Moi Hoon Yap. Spotting macro-and micro-expression intervals in long video sequences. In 2020 15th IEEE International Conference on Aulomatic Face and Gesture Recognition (FG 2020), pp. 742-748, 2020. doi: 10. 1.110//FG478802220000000
Wenhao Leng, Sirui Zhao, Yiming Zhang, Shiifeng Liu, Xinglong Mao, Hao Wang, Tong Xu, and Enhong Chen. Abpn: Apex and boundary perception network for micro- and macro-expoe exprrssxpnnsno spotting. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, pp. 7160-7164, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392037. doi: 10.1145//503161.33555599 URL 3553161.3351155
1145/
Jingting Li, Zizhao Dong, Shaoyuan Lu, Su-Jing Wang. Wen-Jing Yan, Yinhuan Ma, Ye Liu, Chang- bing Huang, and Xiaolan Fu. Cas(me)3: A third generation facial spontaneous micro-expression database with depth information and high ecological validity. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):2782-2800, 2023a. doi: 10. 2009
Jingting Li, Catherine Soladié, and Renaud Séguier. Local temporal pattern and data augmenta- tion for spotting micro-expressioons IEEF Transactions on Affective Computing, 14(1):81 1000)11-8000 1-822, 2023b. doi: 10. 10.11090TAFFC.220003333333
Xiaobai Li, Xiaopeng Hong, Antti Moilanen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, and Matti Pietikäinen. Towards reading hidden emotions: A comparative study of spontancous micro- expression spotting and recognition methods. IEEE Transactions on Affective Computing, 9(4): 563-577, 2018. doi: 10. 21109TAAFFC.200.2666666.
10

---

Published as a conference paper at ICLR 2024
Gen-Bing Liong, John See, and Lai-Kuan Wong. Shallow optical flow three- for macro- and micro-expression spotting from long videos. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 2643-2647, 2021. doi: 10. 1O09/CCIP42222222219.50000
Haiqi Liu, C. L. Philip Chen, Xinrong Gong, and Tong Zhang. Robust saliency-aware distillation for few-shot fine-grained visual recognition. IEEE Transactions on Multimedia, pp. 1-14, 2024. doi: LLL100/TMM.222443399990
Yong-Jin Liu, Jin- Kai Zhang, Wen-Jing Yan, Su-Jing Wang, Guoying Zhao, and Xiaolan Fu. A main directional mean optical flow feature for spontaneous micro-expressiin recognition. IEEE Transactions on Affective Computing, 7(4):299-310, 2016. doi: 10.1 10.11999FFF..221
Chen Ma, Li Chen, and Junhai Yong. Au r-cnn: Encoding expert prior knowledge into r-cnn for action umit detection. Newrocomputting, 355:35 2019. ISSN 0925-2312. doi: https://ooi.or// n0.10066j.ecccomm221000...00
/wwtseeeeww.cieeececieee.t.. www.rreeeetcceeeeeeeeeeeeeeeeeee scienc//
aariiclle//11/5022222212223
Maphox. pixelmatch 0.3.0.
2022. Accessed:
2023-08-01.
Fangbing Qu, Su-Jing Wang, Wen-Jing Yan, He Li, Shuhang Wu, and Xiaolan Fu. Cas(me)² A database for spontaneous macro-expression and micro-expressinn spotting and recognition. IEEE Transactions Affective Computing, 9(4):424 436, 2018. doi: 10. 11110TAFFC..201..654444
Abraham. Savitzky and Mje Golay. Smoothing and differentiation of data by simplified least squares procedures. Analytical Chemistry, 36(8):1627-163)) 1964.
Mingxing Tan, Ruoming Pang, and Quoc V. Le. Efficientdet: Scalable and efficient object de- tection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi: 10. http:////442200022.00000000 42600 2020. 01079
URL. http: dei .rgg/10.1099/
Thuong-Khann Tran, Quang Nhat Vo, Xiaopeng Hong, and Guoying Zhao. Dense prediction for micro-expression spotting based on deep sequence model. In IMAWM, 2019. URL https:
//api. www.ntetttttt.....mm............ hwtpraatttcccccolaaaoooorrrrrrpt
2040793381
Thuong-Khanh Tran, Quang-Nhat Vo, Xiaopeng Hong, Xiaobai Li, and Guoying Zhao. Micro-
expression spotting: A new benchmark. Neurocomputing, 443:356-368, 2021. ISSN 0925-2312.
doi: o0tmr///..aacccrrrrre///////2230 com/
ht wwtp://wwwww.... http://www.iiieeedeeett.tt
Michiel Verburg and Vlado Menkovski. Micro-expression detection in long videos using optical flow and recurrent neural networks. In 2019 14th IEEE International Conference Automatic Face & Gesture Recognition (FG 2019), pp. 1 1-6, 2019. doi: 10.1109/FG.2219.885555
Su-Jing Wang, Ying He, Jingting Li, and Xiaolan Fu. Mesnet: A convolutional neural network for spotting multi-scale eiprossxnression intervals in long videos. IEEE Transactions on Image Jan 2021. doi: 10. 20110/0p.2221..0000 Processing, pp. http://.pp.222...... 3064258 dol.org//o..10000 1109ttt.............
URL http://dx.
Ke Xu, Kang Chen, Licai Sun, Zheng Lian, Bin Liu, Gong Chen, Haiyang Sun, Mingyu Xu, and Jianhua Tao. Integrating videomae based model and optical flow for micro- and macro- expression spotting. In Proceedings of the 31st ACM International Conference Multime- dia, MM '23, pp. 9576-95588 New York, NY, USA, 2023. Association for Computing Machin- ery. ISBN 9798400701085.
doi: 10.1145/358178336622888
https:tts://oi.cooo wtt.doi.oog/1...
1145/358 00455/551783..0 787.7733....83. 3612868
Shiting Xu, Zhiheng Zhou, and Junyuan Shang. Asymmetric adversarial- feature disentangle- ment learning for cross-database micro-expresssion recognition. In Proceedings of the 30th ACM International Conference Multimedia, MM '22, pp. 5342-5350, New York, NY, USA, 2022. Association for Computing Machinery, ISBN 978 9781450392037. doi: 10.1145/350316..333 35544433
URL URL wttp:tttt////c //do http:///dd..ooo1.1111//5//////// 354844435

---

Published as a conference paper at ICLR 2024
Luoyang Xue. Ting Zhu, and JingSong Hao. A two-stage deep neural network for macro- and micro-expression spotting from long-term videos. In 2021 14th International Symposium on Computational Intellignnce and Design (ISCID), pp. 282,286, 2021. doi: 10. 10.1109//SCIDS2799. 2021.00072.
Wen-Jing Yan, Qi Wu, Jing Liang, Yu-Hsin Chen, and Xiaolan Fu. How fast are the leaked facial expressions: The duration of micro-expressiios. Journal of Nonverbal Behavior, pp. 217-230. Dec 2013. doi: 10. 0011007/110119-013-0195549 URL http: / wwt..oo/i.og......o
Wen-Jing Yan, Xiaobai Li, Su-Jing Wang, Guoying Zhao, Yong-Jin Liu, Yu-Hsin Chen, and Xiaolan Fu. Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PLOS ONE. 9(1): 8, 01 2014. doi: 10. 10.1371/jone /o.137//jormmal.ooo 0086041.. URL nattps://dot
org org/10.1371 1ww.1//7//oouaaa. pone 0086041
Yang, Jianming Wu, Zhiguang Zhou, Megumi Komiya, Koki Kishimoto, Jianfeng Xu, Keisuke Nonaka, Toshiharu Horiuchi, Satoshi Komorita, Gen Hattori, Sei Naito, and Yasuhiro Tak- ishima. Facial action unit-based deep learning framework for spotting macro- and micro- expressions in long video sequences. In Proceedings of the 29th ACM International Con- ference on Multimedia, MM '21, pp. 4794-4798, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386517. doi: 11144//4474085.4772299 URL 347929
Le Yang, Houwen Peng, Dingwen Zhang, Jianlong Fu, and Junwei Han. Revisiting anchor mecha- nisms for temporal action localization. IEEE Transactions Image Processing, pp. 8535 8548, Jan 2020. doi: 200000990 URL http:/ dx. /ttp://222......r//11111////////
20200 30164861
Chuin Hong Yap. Connah Kendrick, and Moi Hoon Yap. Samm long videos: A spontaneous facial micro- and macro-exprsssions dataset. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 771-776, 2020. doi: 10. 10.1109/FG47889.22222
00029.
Chuin Hong Yap. Moi Hoon Yap, Adrian K. Davison, Connah Kendrick, Jingting Li, Sujing Wang, and Ryan Cunningham. 3d-cnn for facial micro- and macro-expression spotting on long video sequences using temporal oriented reference frame, 2022.
Shukang Yin, Shiwei Wu, Tong Xu, Shifeng Liu, Sirui Zhao, and Enhong Chen. Au-aware graph convolutional network for macroand micro-expression spotting. In 2023 IEEE International Con- ference on Multimedia and Expo (ICME), pp. 228 228-233, 233, 2023. doi: 10. 10//C//555 1.2023. 00047.
Wang- Wang Yu, Jingwen Jiang, and Yong-Jie Li. Lssnet: A two-stream convolutional neural network for spotting macro- and micro-expresssion in long videos. In Proceedings of the 29th
ACM International Conference on Multimedia, Oct 2021. doi: 00.114533440085.47992 URL
http://dks http://ee..cco.ooro./////////653 34792115
He Yuhong. Research micro-expression spotting method based optical flow features. In
Proceedings of the 29th ACM International Conference on Multimedia, MM '21, pp. 4803-4807, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 9781450386517, doi:
10. 041145/3474085.3479222
URL //dottt/p//.o..... 100 010.1114111111410....1.. 04774448888744799
C. Zach, T. Pock, and H. Bischof. duality based approach for realtime tv-11 optical flow. Pro- ceedings of the 29th DAGM Conference Pattern Recognition, pp. 213-223, Berlin, Heidelberg, 2007. Springer-Verlag, ISBN 9783540749332.
Wei Zhang, Jingting Li, Su-Jing Wang, Xian-Hur Duan, Wen-Jing Yan, Hai- Yong Xie, and Shu- Cheng Huang. Spatio-tempont fusion for macro- and micro- expression spotting in long video sequences. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recog- nition (FG 2020), pp. 734-741, 2020. doi: 10. D0.11099FG478882222000000
12

---

Published as a conference paper at ICLR 2024
Yi Zhang, Xinhua Xu, Youjun Zhao, Yuhang Wen, Zixuan Tang, and Mengyuan Liu. Facial prior guided micro- exprrs-eppesssinn generation. IEEE Transactions on Image Processing, 33:525-540, 2024. doi: 10.1109911P22033335177..
Zongyan Zhang, Haohan Weng, Tong Zhang, and C. Philip Chen. A broad generative network for two-stage image outpainting. IEEE Transactions on Neural Networks and Learning Systems, 200.9TTNNSSS.223322242....
pp. 1-15, 2023. doi:
Yuan Zhao, Xin Tong, Zichong Zhu, Jianda Sheng, Lei Dai, Lingling Xu, Xuehai Xia, Yu Jiang, and Jiao Li. Rethinking optical flow methods for micro-expression spotting. In Proceedings of
the 30th ACM International Conference on Multimedia,, MM '22, pp. 7175-7179, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392037. doi: 10.1145/ 3503161.3551602. URL /tt.a/tt.//......//0000113311111
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PSEUDO-CODE FOR FACIAL ALIGNMENT
The proposed facial alignment algorithm consists of Alignment and Reference Frame Update. Python package Pixelmatch (Mapbox 2022) provides function pizelmatch to calculate the pixel difference between two images and return mismatched pixel number m. We choose the nose gion the matching box, and it is cropped according the center coordinate, which is why there are inputs function pirelmatch. The alignment adjusting the center coordinate 6; of cur- rent frame, until the difference between the current matching box and the reference match box is minimal. The process of adjustment is based on the ideology of greedy algorithm, selecting the direction in which decreases the most the current adjustment direction until is minimized.
Algorithm Alignment
Input: current frame and center fc, reffereencce frame and center JR, CR Output: center coordinate ci after alignment, mismatch m directions = [[0, 0j. [], [0, [-1, 0]. -1]] fn fo, G. CR, update 1, listm x5
m mismatch( fi, fn, C. Cn)
m < Minn then
return Ca,
end if
while update * do
for i = 0 to len direcction)) do
list pixclmatch(fi, fa, c, + directions(i], ca)
end for update- argmin(list..) G G + directions (update]
end while return list. [0]
The head posture or facial appearance is prone to change over a prolonged period of time, which why we update the RF dynamically. However, it is critical guarantee that the updated RF is perfectly aligned to ensure subsequent frames do not deviate. The cases of immediate update in- clude
and m > Mmaz. indicating perfect alignment and large posture change respectively. Tau is set for delayed update, and it can be adjusted according to the FPS (Frames per second) of the dateset. To reduce jitter, count increases only when the center coordinate of current cutting box differs from the center coordinate of previous cutting box.
Algorithm Reference Frame Update
Input: current frame center ft, c.rrferencce frame and center JR CR Output: updated reference frame and center JR, CR JR. = fo.
on. cound 0, mTt. inf, smp.. Ctmp
e, Mi, Alignment( /2. JR.c,, CR)
m.
m, > Mmaz then
fR = fa, CR Ca count 0, m, = inf
else
if count < Tan, then if S m, then
fomp = fi, Comp = Co. m, = m.
end if
if * c. then count count
end if
else
fR famp, CR = Ctmp, count = 0, m. = inf
end if
end if
return fR. CR
14

---

Published as a conference paper at ICLR 2024
A.2 VALIDATION OF M M-PATTERN
CASME II Yan et al. 2014). CAS(ME) e Qu et al. 2018) and SAMM-LV Yap et al. 2020 are applied to the validation of M M-pattern. For CASME II, there is only one or zero ME with little head movements in each video. Thus it is perfect dataset for validation of M-pattern. In contrast, the videos in CAS(ME)² and SAMM-LV consist of mutiple type of movements, so we use these two datasets to verify the robustness and distinctiveness of M-pattern. Fig. 11 illustrates the feature curves of MEs on the above three datasets, with the ground-truth between the red lines. It can be observed that the feature curves for MES of various lengths all conform M-pattern.
1.20
214
CUOME WART IPE5
100
322
Figure 11: Feature curves of MES in three datasets.
A.3 DATASETS AND METRICS
CAS(ME)² consists of 22 participants and 98 long videos, including 300 macro-expressions and 57 micro-expressions.
SAMM-LV consists of 32 participants and 147 long videos, including 343 macro-expressiin and 159 micro-expressions in the long videos.
CASME II comprises of 246 micro- micro-expressinn expressions out of 26 participants. Compared to the two datasets above, each video is shorter in duration and contains only zero micro-movement.
Table 3 summarizes the three databases.
We follow the performance metrics proposed in MESNet Wang et al. 2021). For spotted interval W wspotted, the prediction is considered as a true positive (TP) when there is a ground truth interval W groundTruth fitting the following condition:
W spottted W groundTruth > 0.5.
(11)
W spottedd W groundTruth
Otherwise, it is false positive(FP). Specifically, there are m ground intervals in dataset and ground-truth intervals spotted; and if there predicted intervals with b TPs. Then the recall,
15

---

Published as a conference paper at ICLR 2024
precision and Fl-score computed as follows:
a Recoll = _ Precision b F' - score 2 Recall x Precision
(12)
Recoll + Precision
The final evaluation performed the entire dataset, based on the overall Fl-score of MaE and ME spotting performance.
Table 3: Details of CAS(ME)², SAMM-LV and CASME II
CAS(ME)²
SAMM-LV ME
CASME II
ME
MaE
MaE
ME
Frames per second
30
200
200
Num of samples
57
300
159
343
255
Max duration
17
118
101
5175
141
Min duration
9
4
30
102
24
N
14
36
74
259*
67
k
7
18
37
130*
34
For N. SAMM-LV, samples over frames are omitted.
A.4 EXPERIMENT SETTINGS
We spot ME and MaE respectively using the proposed methods. TV-LI et al. 2007) is used for basic OF calculation. is set to 3 for peak detection. For MES, T Tmaz = [23, 51). Tddeg {45, 250), !oUmerge = {0.3, . CAS(ME)² and SAMM-LV respectively). The MaE in datasets do not have discriminatory features like ME, thus we only set minimum threshold {15,9) 9} to exclude movements with lower ddeg for MaE spotting. In feature extraction phase, Tateorder block area where two refer to bins in total and the scaling factor respectively. Savitzky- Golay filterSSaiizzk & Golay] 1964) is used to smooth the feature curve, where window length and polyorder is set to & and 3. For facial alignment algorithm, frame with lower than Mmin will not execute Alignment than Mmaz is likely to have a significant change in head posture facial appearance. Mmin and Mmax is set to 0.1% and 8% of the matching box area respectively.
8x8
order to reduce jitter between frames; frame with my larger
A.S EXPERIMENTAL RESULTS IN DETAIL.
Detailed spotting results. Table shows the detailed spotting results. 'Find' refers the number of spotted ME/MaF, while "TP' refers to the number of predicted intervals that satisfy Eq. (11). Our approach drastically reduces the number
about half of the total number of expressions (ME and MaE). Moreover, almost all FP were generated by habitual movements and no global movement wrongly detected as ME/ MaE.
Table 4: Details of the spotting results.
CAS(ME)
SAMM-LV
CASME
ME
MaE
All
ME
MaE
All
ME
Total
57
300
357
159
343
502
255
Find
20
140
160
110
157
225
TP
20
152
172
47
111
158
225
FP
123
199
122
139
261
45
Precision
0.2083
0.5527
0.4636
0.2781
0.4440
0.3771
0.8333
Recall
0.3509
0.4667
0.4482
0.2956
0.3207
0.3127
0.8823
F1-score 0.2614
0.5061
0.4558
0.2866
0.3724
0.3419
0.8571
Table 5 provides the progressive results of spotting rules. Since the number of TP before merging far exceeds the number of ME, we adopt a new metric for better comparison. Compared to Eq. (12), the formula for Precision modified to a Practically, the occurrence and restoration phases of ME
16

---

Published as a conference paper at ICLR 2024
not strictly symmetrical, hence two peaks in M-pattern not equal in most cases. To prevent the smaller peak in M-pattern from being omitted, we set lower threshold in step 1. Thus the initial number of FP is large. Leveraging the properties of ME (step 3, 6) helps filter out a large number of FP. The merging (step 5) ensures that the merged interval is not shifted to the left or right and the length of merged interval is almost equal to the average length of intervals involved in merging. Only MEs in SAMM-LV are excluded after merging for they no longer satisfy Eq.(1
Table 5: Progressive MES results in detail.
CAS(ME) ME
SAMM-LV ME
Step
Find
TP
FP
Fl-score
Find
TP
FP
FI-score
1,
32
74
2259
0.0600
88
319
3938
0.0399
1, 2, 1, 2, 3,
22
35
293
0.1672
178
934
0.0803
22
31
240
0.1765
50
175
838
0.0853
1,
2, 3, 4 5
22
22
150
0.1921
48
48
239
0.2152
1, 2,
3,
4, 6
20
20
76
0.2614
47
47
122
0.2866
In the absence of block-wise MDMO, the global movements can only be removed by spotting step 4. But the magnitude of some global movements are too small to be detected under skip-k-frame strat- egy. Therefore, shown in Table 6, the experimental results using MDMO as feature descriptor are worse than block-wise MDMO, mainly due to lower precision compared to the latter. We com- pare different divisions of the OF field: intersecting division in Fig. 6(c) and disjoint division in Fig.7. For MDMO that only require magnitude feature, the difference between two divisions is significant. Whereas for block-wise MDMO, which require directional information, the division proposed in this paper is noticeably superior to the disjoint division. Because both divisions average the OF vectors the main direction, they are close in magnitude and differ more in direction. The proposed intersecting division method has more accurate description in direction, movement in any direction can ultimately be assigned to one bin.
Table Ablation experiments.
CAS(M
SAMMLV
ME
MaE
ME
MaE
MDMO
0.212
0.456
0.250
0.354
MDMO
0.194
0.473
0.245
0.351
MDMO, w/o step3, 6
0.155
0.431
0.197
0.345
MDMO, w/o step4
0.180
0.455
0.249
0.352
MDMO, w/o FA
0.189
0.433
0.224
0.346
MDMO, w/o [FA, step4]
0.180
0.439
0.222
0.343
bwMDMO
0.261
0.506
0.287
0.372
bwMDMQ*
0.200
0.486
0.241
0.345
MDMO: All frames treated as order state, i.c., Tais is set to 0.
bwMDMO: Block-wise MDMO. FA: Pixelmatch- facial align- ment. * Original partition of bins(Fig. 7).
17


Reviews:
[{"review_id": "7vEmSnZUQu", "reviewer": "Reviewer_o3nk", "summary": "The paper introduces a novel mean-optical-flow (MDMO) feature for Micro-expression spotting, along with a series of spotting rules on the M-curve yielded by such features.\nThe authors were successful in showing how Pixelmatch-based facial alignment followed by MDMO feature extraction are effective to hamper disturbances from global movements that greatly impact the performance of MES methods. They also provided enough evidence of the effectiveness and soundness of the proposed spotting rules.\nThe authors claim to have outperformed SotA methods and provide numerical results to sustain their claim.", "questions": "- Is there any particular reason to present the feature extraction procedure before describing the M-pattern? (Of course, I can have misunderstood the order of the steps in the method and feature extraction foes last indeed.)\n- You get a unique M-curve for the whole sequence of is one M-curve per ROI?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors were successful in clearly describing the importance of the MES task, the state of the art and, more importantly, the limitations of the field, from a technical point of view.\nThe proposed method, in my opinion, is adequate to sidestep such limitations and the experimental setup. Both MDMO and the M-curve analysis are interesting choices, the the experimental setup, including the comparison to related methods and ablation studies, is clearly well designed to not leave gaps when justifying both the effectiveness of the method, and the design choices.", "weaknesses": "They are exclusively related to the presentation of the paper.\nIn first place, I think the order of Section 3.4 should be before Section 3.2, to match the order of the steps inside the pipeline.\nOn the other hand, the spotting rules are presented in a really verbose manner, making Section 3.3 denser than it should be. I think that the authors can rather present the details in a graphical way (probably a flowchart) and dedicate the main text to describe the decision pipeline in more plain words.\nFollowing the guidelines of other top-tier conferences, I think it is necessary the authors dedicate a few lines on ethical concerns implied in automating MES spotting. This field is particularly sensible from a bias and fairness perspective.\nFinally, I spotted some typos:\n\nPage 3\n- Alignment is the process of adjusting the center coordinate until m between the current matching box and the reference matching box is minimized.\n- to prevent subsequent frames from being aligned with misaligned frame\n- even without facial alignmen.\n- Thus, we propose a spotting rules", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5C9kYg1tPP", "reviewer": "Reviewer_yjCN", "summary": "The paper presents a skip-k-frame strategy for micro expressions spotting in videos. They present a Main Directional Mean Optical Flow (MDMO) feature that is capable of removing global movements from feature extraction stage. Experimental results were presented on three popular datasets and comparison against state of the art methods.", "questions": "On a minor note I have following suggestions to improve the paper:\n- In section 3.1 Face Alignment, how do you determine the values for M_min and M_max?\n- Is it possible to show one visual example of elimination of global movements from feature extraction stage?\n- Some of the graphs are very hard to read in a printed paper as the font size is so small, for example Figures 9 and 10", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Elimination of the global movements in the early feature extraction stage contributes to better detection of micro expressions.\n- The paper is well written and the proposed methodology seem to be technically sound\n- The experimental results show that the proposed method mostly outperformed the state of the art methods", "weaknesses": "I did not find much weaknesses in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zNjJfxPxJs", "reviewer": "Reviewer_Z3Ax", "summary": "In this paper, the authors address the challenging task of micro-expression(ME) spotting, which is difficult due to its subtle nature and other factors, such as head rotation, a global movement during an ME generation. A novel approach called skip-k-frame block-wise main directional mean optical flow (MDMO) feature is proposed by the authors for ME spotting. This is achieved using unfixed reference frames compared to existing work utilizing fixed reference frames. The paper introduces the readers to M-pattern, a distinct movement pattern and enhanced spotting rules based on M-pattern and ME characteristics. The article then presents block-wise MDMO features to remove global movements without compromising ME movements in the feature extraction stage. Experimental results on state-of-the-art datasets demonstrate that the proposed methods outperform existing methods in ME spotting.", "questions": "The reviewer would like to get feedback on the following:\n1. The k value in skip-k-frames was calculated as (N+1)/2, where N is the average ME duration. How does N (average value) influence the spotting of MEs that fall away (beyond a threshold) from this mean? Ex. In Fig 11, was the spotting of specific MEs easier than others (CASME II_sub11 type compared to CASME II_sub01 type)?\n\n2. It was unclear to the reviewer if the authors updated values of k across datasets (based on the average ME duration calculated from the chosen sample set).\n\n3. If indeed k was updated as per the sample set, how would this impact its update in the case of very long videos (which were omitted by authors of this work)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper highlights the strength of recognizing patterns exhibited by actions and further utilizing characteristics exhibited by those actions in spotting them effectively. The authors have validated this understanding in the domain of ME spotting. The proposed work is an improvement on the existing work in the literature.", "weaknesses": "Even though it is a well-written paper, the draft contains many typos. It is of utmost importance that these are fixed. \n\nSome aspects were unclear to the reviewer and are listed under questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YokFzPniVX", "reviewer": "Reviewer_KsQW", "summary": "This paper reported a “M” shaped pattern of optical flow magnitude curve that could be used in micro-expression spotting. Based on the M-pattern assumption, the author proposed a three stage, rule based system to estimate micro-expression interval. The author further compared the proposed method with other micro-expression spotting methods and obtained favorable performance on public dataset.", "questions": "1. For different type of micro-expression spotting, does the system need different set of parameters? \n2. Does the dataset contain a validation part for parameter selection, and a separate test part for metric reporting?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The proposed method has superior performance on important public datasets compared to several other micro-expression spotting methods.", "weaknesses": "1. While the proposed system has many parameters, it is not clear how to choose the parameter value. \n2. There isn’t necessary discussion on the type of micro-expression presented in the dataset, as well as how well the proposed M-pattern fit in each category. \n3. In general I think the structure of the paper could be improved. For example, putting dataset description in appendix makes the experiment section harder to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "omACR5rtyj", "reviewer": "Reviewer_32Bs", "summary": "This article focuses on the issue of global movements affecting the recognition of subtle micro-expressions in micro-expression analysis. It identifies a distinctive motion pattern in micro-expressions, referred to as the \"M pattern,\" and introduces the feature extraction method \"skip-k-frame block-wise main directional mean optical flow (MDMO)\" based on this M pattern. And propose a dynamic update facial alignment algorithm based on pixel matching with unfixed reference frames. The proposed methods are superior to the state-of-the-art methods.", "questions": "The method in the article appears to establish a set of conditions and rules, and I am skeptical about whether this is a universally applicable approach in this field. Is it possible to demonstrate through experimentation that this method exhibits generalizability on other datasets?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This article identifies a unique motion pattern (M-pattern) in micro-expressions and proposes a feature extraction method based on the M-pattern that promises to address the effects of global motion.\n2. Compared to other methods discussed in the article, this approach has achieved the current state-of-the-art results in Micro-Expression Spotting tasks and provides partial explanations for the issues it aims to address.", "weaknesses": "1. No more detailed theoretical analyses of why the M-pattern is effective in mitigating the influence of other global movements. It also does not offer the necessary alternative modes or direct ablation experiments involving muscle intensity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)