Background• Cross-modal Retrieval• Metric:• Flickr30K: Recall@K Bi-directional• MSCOCO: Results on 1000 test images and their corresponding sentences• Continual Learning• Other names: lifelong learning, sequential learning or incremental learning• Key problem: catastrophic forgetting (CF) of old concepts as new ones learnt• Learning representations for a new domain (called a task)

Introduction• Continual learning + Cross-modal Retrieval ?
• Retrieval: Traning -> Indexing -> Query• Pay special attention to the role of “indexing” stage• Contribution:• A continual cross-modal retrieval framework• Identify and study the different factors lead to forgettting in cross-modal
embeddings and retrieval• Study modifications in the retrieval framework, network archi. and regularization

Continual Cross-modal Retrieval• Cross-modal Deep Metric Learning• Two-branch network: image-specific & text-specific• Aligned with similarity matrix S (binary)• Constraints:• x: image• y: text• Loss:Continual Cross-modal Retrieval• Training, indexing and query stages:• Training: Learning embedding networks• Indexing:• Construct a database expressed with embeddings• Training data are not necessarily same as indexing data
• For simplicity, consider they are the same

• Query: Compute similarity between a query sample and the index data
• Deplyed system only conduct queryContinual Cross-modal Retrieval• Continual Learning in Retrieval• Setting:• Data are presented as a sequence of tasks• Each task involves data from a different domain (animal, vehicle ...)• Embedding networks are updated• Evaluation with seperate data from each task
• Classify a negative pair as intra-task neg. pair (ITNP) & cross-task neg. pair (CTNP)• CNTPs are not available during training• Assume all positive pairs are intra-task

Continual Cross-modal Retrieval• Continual Retrieval:• Reindexing or not?
Reindexing•••No Reindexing

Index both data of current and•Only index data of current task

previous tasks•Efficient

Database and query are processed•Asymmetry, query embeddings arewith the same network

extracted with new operators while

Time & Resourse consuming

database embedding are extractedwith old ones

Catastrophic Forgetting• Embedding networks:• Parameters drift from priviously
optimal values• Embedding misalignment:• Embeddings of different modalities
may drift differently• Task overlap:• CTNPs are the only force todiscriminate samples of differenttasks

Preventing Forgetting• Embeddding drift• Regularization term: To penalize weighted Euclidean distance• Θ and Ω are iteratable weights (initialized as 0)• Methods to iterate are left out• Final loss:

Preventing Forgetting• Unequal Drift• Tying the networks by sharing layers at the top• Bottom layers must remain modality-specific• Decoupling Retrieval directions• In the case of no reindexing• Beneficial when image and text embeddings drift in different directions• Cross-task overlap• Weight regularization and sharing layers could help

Experiments• Settings:• Joint vs Continual• I2T & T2I• Known task & Unknown task
• Reindexing• Weight regularization• Decoupled directions• Layer sharing

Experiments• Sequential Visual Genome (SeViGe)• Divide Visual Genome into three domains: animals, vehicles and clothes

Experiments• Sequential MS-COCO (SeCOCO)• Challenging to organize data into tasks

Conclusion• A piece of “digging hole” work
• Massive experiments• Lack of dataset and “tasks”