Question: The main issues of paper are the experimental results. In table 1, NCCL + Reservoir and NCCL + Ring underperforms ER-Ring, could the authors give some discussion on this?
Evidence: The following tables show our main experimental result, which is averaged over 5 runs. We denote the number of example per class per task at the top of each column. Overall, NCCL variants outperforms baseline methods especially in the forgetting metric. Our goal is to demonstrate the usefulness of the adaptive learning rate scheme to reduce the catastrophic forgetting, and verify the proposed theoretical convergence analysis. We remark that our adaptive learning rates successfully suppress forgetting by a large margin compared to baselines. Note that NCCL also outperform A-GEM, which does not maximize transfer ∇f It (x t ), ∇g Jt (x t ) > 0. Now, we can empirically demonstrate our theoretical guaranteed method by minimizing Γ t is valid.
We clipped β Ht to increase the performance. As we discussed earlier, we can prevent forgetting when ∇f It (x t ), ∇g Jt (x t ) > 0. However, we observe that ∇f It (x t ) 2 suddenly increases because of the interference at the previous step t − 1. The very large learning rate β Ht by the increased ∇f It (x t ) can force the model to fall into an arbitrary point that is likely to increase the loss of f . Clipping the learning rate reduces this problem and still has the effect of reducing the catastrophic forgetting term Γ t . By the property of quadratic polynomial, the catastrophic forgetting term is negative because the clipped value is smaller than the original learning rate.
We show that NCCL is a potentially powerful alternative for continual learning. Even with tiny replay memory, NCCL still performs better than some baselines. We note that NCCL shows the best performance on the forgetting metric. It implies that NCCL prevent catastrophic forgetting more efficiently than others by minimizing the catastrophic forgetting term in the proposed optimization problem. However, the accuracy is slightly lower than other baselines, which include experience replays. The purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring, stable-SGD, and ORTHOG-subspace. This result shows that the plasticity to learn a new task is restricted by NCCL variants with tiny memory. In particular, we would expect that NCCL would benefit from the additional enhancements in ORTHOG-SUBSPACE and stable SGD by introducing their techniques. In appendix, we add more results with larger sizes of memory, which shows that NCCL outperforms on the average accuracy. We conclude that the transfer effect by the small size of memory for NCCL is less effective.
The following tables show our main experimental result, which is averaged over 5 runs. We denote the number of example per class per task at the top of each column. Overall, NCCL variants outperforms baseline methods especially in the forgetting metric. Our goal is to demonstrate the usefulness of the adaptive learning rate scheme to reduce the catastrophic forgetting, and verify the proposed theoretical convergence analysis. We remark that our adaptive learning rates successfully suppress forgetting by a large margin compared to baselines. Note that NCCL also outperform A-GEM, which does not maximize transfer ∇f It (x t ), ∇g Jt (x t ) > 0. Now, we can empirically demonstrate our theoretical guaranteed method by minimizing Γ t is valid.
We clipped β Ht to increase the performance. As we discussed earlier, we can prevent forgetting when ∇f It (x t ), ∇g Jt (x t ) > 0. However, we observe that ∇f It (x t ) 2 suddenly increases because of the interference at the previous step t − 1. The very large learning rate β Ht by the increased ∇f It (x t ) can force the model to fall into an arbitrary point that is likely to increase the loss of f . Clipping the learning rate reduces this problem and still has the effect of reducing the catastrophic forgetting term Γ t . By the property of quadratic polynomial, the catastrophic forgetting term is negative because the clipped value is smaller than the original learning rate.
We show that NCCL is a potentially powerful alternative for continual learning. Even with tiny replay memory, NCCL still performs better than some baselines. We note that NCCL shows the best performance on the forgetting metric. It implies that NCCL prevent catastrophic forgetting more efficiently than others by minimizing the catastrophic forgetting term in the proposed optimization problem. However, the accuracy is slightly lower than other baselines, which include experience replays. The purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring, stable-SGD, and ORTHOG-subspace. This result shows that the plasticity to learn a new task is restricted by NCCL variants with tiny memory. In particular, we would expect that NCCL would benefit from the additional enhancements in ORTHOG-SUBSPACE and stable SGD by introducing their techniques. In appendix, we add more results with larger sizes of memory, which shows that NCCL outperforms on the average accuracy. We conclude that the transfer effect by the small size of memory for NCCL is less effective.
Overall, NCCL variants outperforms baseline methods especially in the forgetting metric.****We show that NCCL is a potentially powerful alternative for continual learning. Even with tiny replay memory, NCCL still performs better than some baselines. We note that NCCL shows the best performance on the forgetting metric. It implies that NCCL prevent catastrophic forgetting more efficiently than others by minimizing the catastrophic forgetting term in the proposed optimization problem. However, the accuracy is slightly lower than other baselines, which include experience replays. The purpose of our adaptive learning rate scheme is to prevent catastrophic forgetting, so the performance of current task is slightly lower than ER-Ring, stable-SGD, and ORTHOG-subspace. This result shows that the plasticity to learn a new task is restricted by NCCL variants with tiny memory. In particular, we would expect that NCCL would benefit from the additional enhancements in ORTHOG-SUBSPACE and stable SGD by introducing their techniques. In appendix, we add more results with larger sizes of memory, which shows that NCCL outperforms on the average accuracy. We conclude that the transfer effect by the small size of memory for NCCL is less effective.

Answer: We note that our method has the smallest forgetting metric among all baselines. It implies that our method is focused on reducing catastrophic forgetting. We think that learning on the current task $C$ is not enough to show the best scores for the average accuracy. However, we emphasize that the results with larger memory outperforms the baelines. Then, we can conclude that our method suffers from the transfer effect with small memory. Because, the small subset cannot cover the gradients for entire datapoints.