Video Object Segmentation (VOS)• Semi-supervised setting• Provide first frame annotation• Multi instancesVideo Object Segmentation (VOS)• Problem with long videos• Sacrificed segmentation quality• Reason: The features of the memory frame arecompressed and the information is lost

• Problem of memory matching• Recurrent approaches: Prone to drifting andstruggle with occlusions (Low Performance);• Attention based: Required large amount of GPU• Contribution of XMem• Architecture (based on Atkinson–Shiffrin memorymodel) that can handle long video through a LongTerm Memory• New Memory Reading technic that can obtain goodsegmentation results while consuming only a smallamount of GPU resources

XMem —— Overview• Inspired by the Atkinson–Shiffrin memory model• Sensory memory: Cues used in decoding each frame• Working memory: Including only a few frames full memory• Long-term memory: Compressed memory of a large amount of frames

XMem —— Overview• Inspired by the Atkinson–Shiffrin memory model• Sensory memory: Cues used in decoding each frame• Working memory: Including only a few frames full memory• Long-term memory: Compressed memory of a large amount of frames

XMem —— Overview• Inspired by the Atkinson–Shiffrin memory model• Sensory memory: Cues used in decoding each frame to improve temporal consistency• Working memory: Including only a few frames full memory• Long-term memory: Compressed memory of a large amount of frames

XMem —— Overview• Inspired by the Atkinson–Shiffrin memory model• Sensory memory: Cues used in decoding each frame• Working memory: Including only a few frames full memory• Long-term memory: Compressed memory of a large amount of frames

XMem —— OverviewEncoder-Decoder PipelineQuery Encoding

Value Encoding(for memory)

XMem —— OverviewEncoder-Decoder PipelineQuery Encoding

Value Encoding(for memory)

XMem —— OverviewShort-termEncoder-Decoder PipelineQuery Encoding

Value Encoding(for memory)

Long-term

Sensory Memory12• Retains low-level information which nicely complements the lack of temporal locality in the working/longterm memory• Hidden feature h of GRU1 is updated in each frame (sensory)• Perform deep update every r-th frame using new memory value with GRU2. Advantages are:• discard redundant information that has already been saved to the working memory;• receive updates from a deep network (i.e., the value encoder) with minimal overhead as we are reusingexisting features.

XMem —— OverviewMemory Reading

Memory Reading• s directly scales the similarity and explicitly encodes confidence• e controls the relative importance of each channel in the key space such thatattention is given to the more discriminative channels

XMem —— Overview

Memory Consolidation

Other Memories• Working MemoryT is the first frame and the last r-1 frames(r = 5)

• Long-Term Memory• Compression 1:(T1~Tt-r)• Compression 2:(Prototype selection),(Memory Potentiation)• Removing Obsolete Features• Introduce a least-frequently-used (LFU) eviction algorithm• Selection is also based on cumulative affinity (similar to Prototype selection) after top-k filtering[1]• TotalLong-Term Memory• Prototype Selection• Pick Top-P frequently used candidates as prototypes• Usage is defined by its cumulative total affinity in W and normalized by theduration that each candidate is in the working memory• Memory Potentiation• Apply channel-wise potentiation to prevent aliasing• The enhancement is achieved by aggregating the affinity pixels of the featuremap, and the calculation formula of the similarity matrix can be reused

XMem —— Overview

Experiments Result on Long Videos

Experiments Result on Short Videos

Ablation Studies