# Temporal Action Detection in Untrimmed Videos from Fine to Coarse Granularity
## 3 Our Method
### Discriminative Temporal Selective Search  
Inspired by the selective search [39] used in RCNN object detection [5], we introduce a temporal selective search to generate window-level action proposal candidates: two adjacent segments with minimum distance are merged as a temporal region, iteratively, until there is only one temporal region, and the regions existed in the merging procedure are output as the action proposal candidates. Furthermore, different with the original selective search that is performed in the non-discriminative feature space, such as color space and text space, our temporal selective search is performed in the discriminative feature space. Each segment is predicted into background or action and its feature is extracted by binary Res3D classifier. An example of the segment-level prediction is illustrated in Figure 4. We denote a segment as an action segment if it is predicted into action by RGB or OF Res3D. We can take advantage of this discriminability of the segments in temporal selective search to improve the quality of the proposal candidates by designing three new principles in the procedure of temporal selective search:  
1. [leftmargin=*]
2. Merge Principle: In each iteration, the merged segment or temporal region should contain action segment.
3. Keep Principle: For a temporal region, if the percentage of predicted action segments is below a threshold \(\theta\), it would be excluded from the set of proposal candidates.
4. Stop Principle: The merging is stopped until there is only one temporal region containing action segments.  
Figure 3: Two consecutive frames, optical flow in the horizontal and vertical directions and 3-channel optical flow.  
We name this proposed method Discriminative Temporal Selective Search, and detail it in Algorithm 1. For a complementary similarity measurement of segments, we use L2 distance on the deep feature of the GRB and OF Res3D architectures. The similarity measurement of two segments \(s_{i}\) and \(s_{j}\) is a combination of two measurements:  
\[sm\big{(}s_{i},s_{j}\big{)}=sm_{RGB}\big{(}s_{i},s_{j}\big{)}+sm_{OF}\big{(}s_{i },s_{j}\big{)}, \tag{1}\]  
The threshold \(\theta\) in the Keep Principle is set to 0.4. We select the temporal regions with length between 2 and 50 segments as the proposal candidates, which are feed into a regression network for the further processing.  
```
Input: the video \(V\) and its segments set \(S=\{s_{i}\}_{1}^{num}\)  discriminative segment-level features set \(F=\{f_{i}\}_{1}^{num}\)  segment-level prediction set \(PR=\{pr_{i}\}_{1}^{num}\) Output: Set of the temporal action proposal candidates \(PC\)  Initialize similarity set \(SM=\varnothing\)  Initialize proposal candidates set \(PC=\varnothing\) foreach consecutive segment pair \(\big{(}s_{i},s_{j}\big{)}\)do  Calculate similarity \(sm\big{(}s_{i},s_{j}\big{)}\); \(SM=SM\cup sm\Big{(}s_{i},s_{j}\big{)}\); while more than one \(s_{i}\) contains action segment do  Get the highest similarity \(sm\Big{(}s_{i},s_{j}\Big{)}=\max(SM)\), where \(s_{i}\) or \(s_{j}\) contains action segment;  Merge \(s_{i}\) and \(s_{j}\): \(sm_{merge}=s_{i}\cup s_{j}\);  Merge the features of \(s_{i}\) and \(s_{j}\): \(f_{merge}=mean\Big{(}f_{i},f_{j}\Big{)}\);  Calculate similarity set \(SM_{merge}\) between \(s_{merge}\) and its consecutive segments or regions;  Update similarity set: \(SM=\Big{(}SM\Big{(}sm(s_{i},s_{*})\cup sm\Big{(}s_{*},s_{j}\Big{)}\Big{)}\Big{)} \cup SM_{merge}\);  Update features set: \(F=\Big{(}SM\Big{(}f_{i}\cup f_{j}\Big{)}\Big{)}\cup f_{merge}\);  Update segments set: \(S=\Big{(}S\backslash\Big{(}s_{i}\cup s_{j}\Big{)}\Big{)}\cup s_{merge}\); if the percentage of action segments in \(sm_{merge}\) beyond threshold \(\theta\)then \(PC=PC\cup s_{merge}\);
```  
**Algorithm 1**Discriminative temporal selective search  
Figure 4: The segment-level prediction by the binary RGB Res3D classifier. The testing video video_test_0000964 is split into \(118\) segments, and each segment is predicted into action or background.