Question: The authors argue that "the adaptive learning rate scheme is to prevent catastrophic forgetting", but the average accuracy maybe is a more important metric？
Evidence: The following tables show our main experimental result, which is averaged over 5 runs. We denote the number of example per class per task at the top of each column. Overall, NCCL variants outperforms baseline methods especially in the forgetting metric. Our goal is to demonstrate the usefulness of the adaptive learning rate scheme to reduce the catastrophic forgetting, and verify the proposed theoretical convergence analysis. We remark that our adaptive learning rates successfully suppress forgetting by a large margin compared to baselines. Note that NCCL also outperform A-GEM, which does not maximize transfer ∇f It (x t ), ∇g Jt (x t ) > 0. Now, we can empirically demonstrate our theoretical guaranteed method by minimizing Γ t is valid.
We clipped β Ht to increase the performance. As we discussed earlier, we can prevent forgetting when ∇f It (x t ), ∇g Jt (x t ) > 0. However, we observe that ∇f It (x t ) 2 suddenly increases because of the interference at the previous step t − 1. The very large learning rate β Ht by the increased ∇f It (x t ) can force the model to fall into an arbitrary point that is likely to increase the loss of f . Clipping the learning rate reduces this problem and still has the effect of reducing the catastrophic forgetting term Γ t . By the property of quadratic polynomial, the catastrophic forgetting term is negative because the clipped value is smaller than the original learning rate.
We show that NCCL is a potentially powerful alternative for continual learning. Even with tiny replay memory, NCCL still performs better than some baselines. We note that NCCL shows the best performance on the forgetting metric. It implies that NCCL prevent catastrophic forgetting more efficiently than others by minimizing the catastrophic forgetting term in the proposed optimization problem. However, the accuracy is slightly lower than other baselines, which include experience replays. The purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring, stable-SGD, and ORTHOG-subspace. This result shows that the plasticity to learn a new task is restricted by NCCL variants with tiny memory. In particular, we would expect that NCCL would benefit from the additional enhancements in ORTHOG-SUBSPACE and stable SGD by introducing their techniques. In appendix, we add more results with larger sizes of memory, which shows that NCCL outperforms on the average accuracy. We conclude that the transfer effect by the small size of memory for NCCL is less effective.
We show that NCCL is a potentially powerful alternative for continual learning. Even with tiny replay memory, NCCL still performs better than some baselines. We note that NCCL shows the best performance on the forgetting metric. It implies that NCCL prevent catastrophic forgetting more efficiently than others by minimizing the catastrophic forgetting term in the proposed optimization problem. However, the accuracy is slightly lower than other baselines, which include experience replays. The purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring, stable-SGD, and ORTHOG-subspace. This result shows that the plasticity to learn a new task is restricted by NCCL variants with tiny memory. In particular, we would expect that NCCL would benefit from the additional enhancements in ORTHOG-SUBSPACE and stable SGD by introducing their techniques. In appendix, we add more results with larger sizes of memory, which shows that NCCL outperforms on the average accuracy. We conclude that the transfer effect by the small size of memory for NCCL is less effective.
Answer: We consider that both average acc and the forgetting metric matters. By considering only average acc, we can underestimate the forgetting by reaching the high performance on the current task. Even if our method shows slightly lower value on average acc, we can observe that our algorithm shows the better perfomance on the forgetting metric.